🐛 FIX: empty lines after certain lists raises exception

[sildar]

This is a minimal fix for #31

Here's a copy/paste of things discussed in the issue for reference:

When processing lists, there is a call to tokenize() that advances the state.line attribute :

list.py 
line 264: state.md.block.tokenize(state, startLine, endLine)

After this line is executed, state.line can be larger than endLine, leading to the IndexError when checking the state at that index.

Breaking right after this line if state.line > endLine doesn't work though, we have to update the nextline variable before, as well as closing the list. It's actually already in the codebase, but one line too late:

line 280 onwards:
    token = state.push("list_item_close", "li", -1)
    token.markup = chr(markerCharCode)

    nextLine = startLine = state.line
    itemLines[1] = nextLine
    contentStart = state.bMarks[startLine]

    if nextLine >= endLine:
        break

The easy solution would be to just put the check a few lines earlier.

line 280 onwards:
    token = state.push("list_item_close", "li", -1)
    token.markup = chr(markerCharCode)

    nextLine = startLine = state.line  # we actually need to update these before breaking

    if nextLine >= endLine:
        break

    itemLines[1] = nextLine
    contentStart = state.bMarks[startLine]

Also, I'm not sure about what itemLines does, it's not really used at all in this method.

I looked at the tokenize() method mentioned at the beginning of this post and I don't think we should modify it.

[codecov]

Codecov Report

Merging #36 into master will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master      #36   +/-   ##
=======================================
  Coverage   95.52%   95.52%           
=======================================
  Files          69       69           
  Lines        3371     3371           
=======================================
  Hits         3220     3220           
  Misses        151      151           

Flag	Coverage Δ
#pytests	95.52% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
markdown_it/rules_block/list.py	98.84% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef59092...602415d. Read the comment docs.

[chrisjsewell]

Thanks @sildar

Firstly, I think it would be good to add an (initially failing) test here to capture the issue.

Secondly, we should bear in mind the code in:
https://github.com/markdown-it/markdown-it/blob/master/lib/rules_block/list.js

contentStart = state.bMarks[startLine] is in the correct place, BUT, JS just returns undefined instead of raising an IndexError, for indices larger than the array. So yeh I think this solution should be fine, or a try/except IndexError block.

Also, I'm not sure about what itemLines does, it's not really used at all in this method.

Its set here, as an alias for token.map:

markdown-it-py/markdown_it/rules_block/list.py

Line 232 in 8f16805

token.map = itemLines = [startLine, 0]

I don't see any obvious reason not to just use token.map, but that it is what's done in the JS code 😬

[chrisjsewell]

I added the test, so I think this is good to go.
Feel free to keep any other PRs coming though 😄 (#8 would be amazing, but it is real head-scratcher!)