Multiline queries are bugged on Windows

[ljani]

To reproduce the issue, try adding the following multiline query to your ledger.beancount and running it:

1970-01-01 query "Food per month" "
select
    year,
    month,
    sum(position) as position
where
    account ~ 'Expenses:Food'
group by
    year,
    month
"

Running the query fails with:

Unknown token: LexToken(error,"\r\nselect\r\n    year,\r\n    month,\r\n    sum(position) as position\r\nwhere\r\n    account ~ 'Expenses:Ruoka'\r\ngroup by\r\n    year,\r\n    month\r\n",1,0)

I think this is an issue with line endings in ledger.beancount. They should be normalized to \n instead of \r\n or the query pipeline should support Windows style line endings as well. Any thoughts?

See also: #764.

[ljani]

Ah, apparenly changing the query to the following fixes the issue:

1970-01-01 query "Food per month" "select
    year,
    month,
    sum(position) as position
where
    account ~ 'Expenses:Food'
group by
    year,
    month
"

However, the first version used to work in 1.15 or so.

Feel free to close the issue if you want, I can live with this workaround.

[yagebu]

Since we get the string from Beancount (the parser) and pass it to Beancount (to run the query) which produces the error, I'd consider this a wontfix in Fava. I'd recommend just using Unix line endings.

[ljani]

I'd recommend just using Unix line endings.

That's exactly the fix I'm looking for in Fava. Editing the ledger file with Fava converts the line endings from \n to \r\n on Windows.

[yagebu]

Editing the ledger file with Fava converts the line endings from \n to \r\n on Windows.

That sounds like a bug then. A PR with a fix would be very welcome.

[ljani]

Ah, apparenly changing the query to the following fixes the issue:

Nevermind. I just noticed something. If I add a new entry using the add transaction dialog, it messes up the line endings. But if I edit the file using the (text) editor, they get corrected back and the queries work.

With this information, it's fairly easy to trace the write to these lines.

Looking at the editor saving the whole source, I traced it to this line, which is using binary mode b as the earlier lines do not do that.

I wonder if Beancount specifies the line endings for the file, or should it support any line endings. In other words, should we fix writing or reading the file?

[ljani]

Based on this:

I'd recommend just using Unix line endings.

We should be specifying the newline='\n' with open() calls as per docs, Stack Overflow and line endings used on these lines. Would this make sense to you?

[yagebu]

I kind of forgot about this one but stumbled upon this in #1758, when naively adding a test that compares the result for ledger.file.get_source() to path.read_text() - the former will not convert any newlines while the latter does.

The current behaviour does not work well for people who want to use a different character from the OS standard like you - we can do better I think by always checking which newline character a file contains and using that to write to those files (falling back to the OS standard if it contains no newline). I'll address that in #1758