Missing end-of-file line feed duplicates paragraph

[gorango]

Changing line 147 in test/index.js to "minify" the incoming html with the following line:

var file = vfile(fs.readFileSync(input, 'utf8').replace(/\n*^\s*/gm, ''))

This particular change results in 14 failures (I recognize that a proper minification might avoid a few of these failures).

I wonder if this issue should be better addressed in the rehype-parse package?

[ChristianMurphy]

@gorango can you provide an example of minified HTML that is causing an issue?

[wooorm]

The regex /\n*^\s*/gm replaces zero or more line feeds, followed by a line feed (important), followed by zero or more white space, with an empty string.
That means that any line feed is removed.

HTML collapses whitespace. That means that a line feed is seen as a single space character.
There’s a difference between nothing and a space.

Sounds like you‘re falling for an XY problem. As Christian mentioned. We need more info.

[gorango]

Agreed 🤦‍♂️. I also didn't take into account the position attributes and WhiteSpaceNodes that are affected by the minification. But just before returning to close this issue, I did spot a small anomaly...

Running rm test/fixtures/**/*.json && node test to generate new output files, I noticed the following results changed unexpectedly:

• test/fixtures/overlap-explicit
• test/fixtures/source

Both outputs contained clones of the last sentence in each.

Adding any space at the end of the vfile.contents prevents the duplicates.

[gorango]

To replicate this, you can replace test/fixtures/source/index.html, with:

<p>This is <span data-nlcst="source">marked</span>.</p><p data-nlcst="source">Completely marked.</p>

You should get similar results if you minify the overlap-explicit/index.html file.

[wooorm]

@gorango I tested these two out and:

• rm test/fixtures/**/*.json && node test did not produce different output.json files for me.
• Replacing test/fixtures/source/index.html with that document, removing its output.json file, and running node test, only changes positional information for me.

Removing the EOF EOL (the last \n before the end of the file) in the source fixture did indeed duplicate the last paragraph.

[gorango]

To clarify, I was running node test with my initial minification hack but I'm glad you were able to reproduce the behaviour.

Duplicates in overlap-explicit could be reproduced just by collapsing the closing root tag:

<section>
 <h1>My Cats</h1>
 <p>You can play with my cat simulator.</p>
 <object data="cats.sim">
  <p>To see the cat simulator, use one of the following links:</p>
  <ul>
   <li><a href="cats.sim">Download simulator file</a>
   <li><a href="http://sims.example.com/watch?v=LYds5xY4INU">Use online simulator</a>
  </ul>
  <p>Alternatively, upgrade to the Mellblom Browser.</p>
 </object>
 <p>I'm quite proud of it.</p></section>

I can confirm that the latest PR handles this case as well.

[wooorm]

Thanks for reporting @gorango, it’s all fixed and released now!

In the future, please do not remove the issue template, instead, but instead take the time to fill it out correctly. It’s there for a reason!