Hard line breaks parsed incorrectly in version 1.1.0

[jamesquilty]

Describe the problem

I found what I think is a bug with the parsing of hard like breaks https://spec.commonmark.org/0.30/#hard-line-break. Example 634

foo\
baz

should be parsed as

<p>foo<br />
baz</p>

but I receive

<p>foo\</p>
<p>baz</p>

The markdown-it demo https://markdown-it.github.io/ produces output consistent with Example 634, so I think this is a bug in markdown-it-py only.

Link to your repository or website

No response

Steps to reproduce

$ markdown-it
markdown-it-py [version 1.1.0] (interactive)
Type Ctrl-D to complete input, or Ctrl-C to exit.
>>> foo\
... baz
... ^D
<p>foo\</p>
<p>baz</p>

The version of Python you're using

Python 3.7.3

Your operating system

macOS

Versions of your packages

No response

Additional context

No response

[welcome[bot]]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.
If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).
Welcome to the EBP community! :tada:

[chrisjsewell]

Heya, thanks. I just tested this locally, and looks to be only an issue with the interactive CLI, i.e. if I write:

test.txt:

foo\
baz

you get

$ markdown-it test.txt
<p>foo<br />
baz</p>

but then with the interactive "mode" I recreate your findings

[jamesquilty]

Thanks for taking the time to reproduce this. It's odd that it only affects the interactive interface, but it's good that it's less serious than if it affected the core code. I'll have to be careful about testing with the CLI in the future!

[chrisjsewell]

Yeh must just be a weird quirk of the CLI stdin, but I'll see if I can work it out at some point

[jamesquilty]

I just tried reproducing the test case with a file and received different results to those you show. I receive:

$ markdown-it test.md
<p>foo\
baz</p>

i.e. the <br /> which should be present at the end of the first line has been rendered as a newline character.

[chrisjsewell]

to check, what version of markdown-it-py and also perhaps check the encoding of the file (although I'm on a mac as well, so should not be a problem)

[jamesquilty]

Well, this is odd: on a different Mac I receive the same output you do. There's a difference in Python version 3.7.3 vs. 3.7.4 and macOS version 10.15.7 vs 10.14, and on one machine I've run pip install mdformat-gfm while on the other I've installed subsequently run pip install --upgrade mdformat without the -gfm. I wouldn't have thought that would affect markdown-it-py.

There's also an odd conflict with pre-commit over importlib-metadata

$ pip install --upgrade pre-commit
...
Requirement already satisfied: importlib-metadata in /Library/Frameworks/Python.
framework/Versions/3.7/lib/python3.7/site-packages (from pre-commit) (4.0.1)
...
Collecting importlib-metadata
  Downloading importlib_metadata-1.7.0-py2.py3-none-any.whl (31 kB)
...
Installing collected packages: importlib-metadata, pre-commit
  Attempting uninstall: importlib-metadata
    Found existing installation: importlib-metadata 4.0.1
    Uninstalling importlib-metadata-4.0.1:
      Successfully uninstalled importlib-metadata-4.0.1
...
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
mdformat 0.7.5 requires importlib-metadata>=3.6.0; python_version < "3.10", but you have importlib-metadata 1.7.0 which is incompatible.
Successfully installed importlib-metadata-1.7.0 pre-commit-2.15.0

which, on the machine which gives the same output as you show, I followed with

$ pip install --upgrade mdformat
...
Collecting importlib-metadata>=3.6.0
  Downloading importlib_metadata-4.8.1-py3-none-any.whl (17 kB)
...
Installing collected packages: importlib-metadata, mdformat
  Attempting uninstall: importlib-metadata
    WARNING: Ignoring invalid distribution -ip (/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages)
    Found existing installation: importlib-metadata 1.7.0
    Uninstalling importlib-metadata-1.7.0:
      Successfully uninstalled importlib-metadata-1.7.0
...
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
virtualenv 20.0.29 requires importlib-metadata<2,>=0.12; python_version < "3.8", but you have importlib-metadata 4.8.1 which is incompatible.
Successfully installed importlib-metadata-4.8.1 mdformat-0.7.9

I've tried pip install --upgrade importlib-metadata as well as uninstalling and reinstalling markdown-it-py on the Mac which is giving the odd output from markdown-it test.txt, but it hasn't resolved that problem.

Otherwise, everything checks-out on both:

$ markdown-it --version
markdown-it-py [version 1.1.0]

$ file -I test.txt
test.txt: text/plain; charset=us-ascii

[chrisjsewell]

Not sure why virtualenv is pinned to this earlier version, but perhaps try using python 3.8, where importlib-metadata is not required. I would certainly recommend using python via a conda installation, rather than the base os's python: https://docs.conda.io/en/latest/miniconda.html

[jamesquilty]

FWIW, I tracked down the problem between machines: trailing white space (invisible!) in the test file on one of the machines originating from, I think, copy-and-paste. The original problem with the CLI remains.