add support for Beancount v3

[yagebu]

This uses beanquery to replace the functionality in beancount.query, which works with both Beancount v2 and v3 - however there are some differences on columns in beanquery compared to beancount.query.

For importers, this sticks to beancount.ingest on v2 and used beangulp for v3. It still expectes importers to conform to the v2 importer protocol for now. Due to changes related to duplicate detection, duplicate detection is not automatically done by Fava here but can still be manually specified as hooks.

[jack9603301]

I look forward to the day when fava and beancount fully support v3. When it enters a stable state, I will immediately migrate from v2 to v3.

Does anyone know the time node?

[blalor]

I've got a query directive like

3000-01-01 query "summary-report" "
    SELECT
        root(account, 2) as account,
        YEARMONTH(date) AS yearmonth,
        SUM(COST(position)) AS balance
    WHERE
        account ~ 'Expenses:'
        AND not (account ~ ':Hold')
        AND currency = 'USD'
        AND date >= 2023-01-01
        GROUP BY 1,2
        ORDER BY 1,2
        PIVOT BY account, yearmonth;
    "

After installing Fava from this branch, I can do bean-query main.beancount .run summary-report successfully, but when running it in Fava I get an internal server error and the following stack trace:

Exception on /personal/api/query [GET]
Traceback (most recent call last):
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/fava/json_api.py", line 174, in _wrapper
    res = func(*validator(data))
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/fava/json_api.py", line 243, in get_query
    return g.ledger.query_shell.execute_query_serialised(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/fava/core/query_shell.py", line 171, in execute_query_serialised
    return QueryResultTable(*serialise_query_result(*res))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/fava/core/query_shell.py", line 234, in serialise_query_result
    tuple(mapper(row[i]) for i, mapper in enumerate(mappers))
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/fava/core/query_shell.py", line 234, in <genexpr>
    tuple(mapper(row[i]) for i, mapper in enumerate(mappers))
          ^^^^^^^^^^^^^^
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/fava/core/query.py", line 154, in serialise
    return simple_units(val)
           ^^^^^^^^^^^^^^^^^
  File "/Users/blalor/Documents/beancount3/.direnv/python-3.12/lib/python3.12/site-packages/fava/core/conversion.py", line 124, in simple_units
    for pos in inventory:
               ^^^^^^^^^
TypeError: 'NoneType' object is not iterable

Otherwise this is looking good!

[yagebu]

After installing Fava from this branch, I can do bean-query main.beancount .run summary-report successfully, but when running it in Fava I get an internal server error and the following stack trace:

Thanks for testing it :) That error (handling a None in an inventory column) should be fixed now - can you retest?

[blalor]

That's working now, @yagebu !

[njhsi]

thanks for the work! how's going? any ETA for release?

[yagebu]

thanks for the work! how's going? any ETA for release?

No ETA for a release which includes this, but I think this is pretty ready for merging yet.

@dnicolodi: I've adjusted for the changes in https://github.com/beancount/beanquery/commit/fe82a6b18016331f90455573e606f4e315303baa. Did you spot any other private API parts that Fava shouldn't be using? Should I add an upper version bound of <0.2 for beanquery to ease iteration there?

[yagebu]

Thanks for the review @dnicolodi :)

[dnicolodi]

Thanks for the review @dnicolodi :)

You're welcome. As you probably realized, I had only a superficial look. I don't pretend to have understood all what the code here does 🙂

[ileodo]

Any plan of releasing this?