fix(app): #4101 Fixes parser to parse new lines directly to output

[mohit-nagaraj]

Fixes parser to parse new lines directly to output.

Closes #4101

Changes

This modification to the addText method ensures that newline characters are properly handled and preserved in the token tree

Checklist

• [ ] Added markup tests, or they don't apply here because...
• [ ] Updated the changelog at CHANGES.md

[joshgoebel]

I think the needed fix here is much simpler. The issue (in my testing) was only when we have an illegal match on $ (end of line) that we then insert no content since technically $ matches an empty string not the \n itself... If you'd like to help you could add a markup test with a problematic sample of this - then I'll throw my simper fix and we'll see if things are still green.

[joshgoebel]

Could you please test https://github.com/highlightjs/highlight.js/pull/4142

[joshgoebel]

If you believe you have test cases that my PR doesn't solve, just point me to them and I'll take another look.

[joshgoebel]

@mohit-nagaraj ping. Could you test my fix against your test cases?

[joshgoebel]

I believe this is closed by https://github.com/highlightjs/highlight.js/pull/4142