bean-add
bean-add copied to clipboard
simon-v
Support for multi-beancount-file setup?
Hi,
as far as I can see, this script does not support a multi-file beancount setup.
I have accounts in one file, settings in one file, and will start to have one transaction file per month. bean-add seems to have issues with this (E.G. tab-completion for accounts does not work at all).
Any plans on improving this?
There weren't. This is a much more complicated feature than one would expect. bean-add wasn't written with this in mind, and implementing it would require some heavy rewrites.
However, there was a planned feature in bean-add that could serve as a workaround for you: writing to external files. With it, you'd make a script such as:
# Make a temporary file to work on
cat accounts.bnct options.bnct transactions.bnct > merged.bnct
# Begin editing that
bean-add merged.bnct
# Upon entering bean-add, activate the "oe" (external write) option
# New transactions are saved into "merged.bnct.new"
# Finish editing
# Append the new transactions to the main file
cat transactions.bnct merged.bnct.new > transactions.bnct.new
mv transactions.bnct.new transactions.bnct
# Clean up
rm merged.bnct merged.bnct.new
...And use that as the wrapper for your work.
Will this be adequate for your purposes?
@simon-v This would be really nice, yes! I even like this approach a bit more, because it keeps bean-add simple, and simple code has less bugs! Wonderful!
I always thought that using the original system (in this case beancount) was the best option for parsing data files. Wouldn't it be better to invoke beancount and get the list of accounts? Then beancount would use whatever features (in this case include files) are currently supported.
Interesting. That kind of option didn't occur to me at the time. Perhaps, i'll try this, with the current system as fallback for edge cases.
Thanks! I'm generally lazy, that's why easy solutions always come to my mind before the hard ones. :)) If you can keep Beancount objects in memory during runtime, that would be pretty much it. And, with picklecache, the performance is generally not that bad. This approach would leave you to only add functionality on top of Beancount instead of rewriting what's already there. I might also have a look once I get to the stage where I need the UI tool more frequently. Cheers!
Actually, the startup delay with beancount in the middle has become quite noticeable. I have used the following command:
time echo q | bean-add journal.bnct > /dev/null
The results are:
With beancount: 0.52s user 0.06s system 98% cpu 0.585 total Without beancount: 0.18s user 0.03s system 98% cpu 0.209 total
That's three times the wait.
With the picklecache removed or invalidated after the journal was changed, the difference is even more significant:
With beancount: 1.83s user 0.06s system 98% cpu 1.920 total Without beancount: 0.18s user 0.02s system 98% cpu 0.208 total
And that's ten times the wait. In fact, i have a feeling that this is going to be the typical user experience (you use bean-add to, you know, add to the journal).
Everything considered, i feel it would be more reasonable to reimplement include, or run bean-query in a background subthread. Both approaches are not without their drawbacks.
With beancount: 1.83s user 0.06s system 98% cpu 1.920 total Without beancount: 0.18s user 0.02s system 98% cpu 0.208 total
Yes, beancount is slow. But IMHO, trying to "quickly" reimplement parsing of beancount syntax is bound to either a) be incomplete and buggy, or b) eventually end up just as complex and slow as the original system.
And that's ten times the wait. In fact, i have a feeling that this is going to be the typical user experience (you use bean-add to, you know, add to the journal).
Simon, I'm not sure I understand the problem you're trying to avoid. IIUC, bean-add is meant to be started once and then used interactively, so the one-time cost of 2s doesn't seem that prohibitive.
FWIW, I have a multi-file ledger with 15+ years worth of transactions (100k+ lines). I can use bean-add by passing in the output of bean-report print (Simple cat doesn't work as the entries need to be sorted for bean-add to be able to do anything useful). But that takes even more than 2s and I, for one, would prefer bean-add to do its own parsing/sorting using the beancount library.
Hi, Pawel, and thank you for your interest in my software. It's worth mentioning that it's basically "dog food", and as such doesn't necessarily fit anyone else's preferred workflow.
With my current journal file, it takes over 3 seconds to load with bean-query, instead of 0.423. It's worth mentioning that i open and close my journal files (plural) several times per day, half of those from a script. As such, the delays compound. If it's not significant for you, i'm happy to tell you that i've added a load-accounts-with-Beancount option to the program in my local git repository (saved in the journal header, default off). Hopefully, this will be good enough for your preferred workflow.
Sounds great! Would you mind pushing that change to your public repository? IIUC, it's flag-protected, so it won't affect existing users.
Done. It's probably worth mentioning that this program has three git mirrors, one at the "canonical" URL, https://simonvolpert.com/bean-add/, one here, on GitHub and one on GitLab. Of those, only the canonical one is guaranteed to be up-to-date.
Today I gave the 'b' option a try. Unfortunately, the results are very different from parsing a bean-report print output (which is what I'm doing normally). The biggest difference is in the recognized currencies, so I added a debug statement to print what it parsed. Looks like with the 'b' option it also lost some 6k records. Just sharing this as an FYI.
$ echo ";*bean-add* b s e d c" > merged.bnct && cat *beancount >> merged.bnct
$ ./bean-add merged.bnct
Loaded options from journal file.
Automatic normalizations applied.
23312 records processed
835 account names, 7358 unique descriptions, 14 currency signs and 90 tags loaded
Initialized currency signs: CHF, USD, CAD, PLN, EUR, -53.28, -560.36, -313.76, USD}, {}, VOO, VGLT, VBR, XAUUSD
Warning: Non-standard indentation.
Changes stay in memory until you [w]rite them out.
Type `h` or `?` for a list of available commands.
$ echo ";*bean-add* s e d c" > merged_sorted.bnct && bean-report main.beancount print >> merged_sorted.bnct
$ ./bean-add merged_sorted.bnct
Loaded options from journal file.
Automatic normalizations applied.
29843 records processed
835 account names, 8642 unique descriptions, 168 currency signs and 90 tags loaded
Initialized currency signs: CAD, USD, PLN, 2008-06-17}, 2008-06-27}, 2008-06-30}, 2008-07-10}, 2008-07-25}, 2008-08-14}, 2008-08-29}, MSFT, 2008-09-11}, 2008-09-26}, 2008-09-30}, 2008-10-16}, 2008-10-31}, 2008-11-14}, 2008-11-28}, 2008-12-15}, 2008-12-19}, 2008-12-31}, 2009-01-02}, 2009-01-12}, 2009-01-30}, 2009-02-04}, 2009-03-31}, 2009-06-30}, 2009-07-13}, 2009-08-31}, 2009-09-30}, 2009-12-31}, 2010-01-12}, 2010-03-31}, 2010-06-30}, 2010-07-11}, EUR, 2010-08-29}, 2010-08-31}, CHF, 2011-01-11}, 2012-05-02}, 2013-05-02}, 2013-06-25}, 2013-12-26}, 2014-04-01}, 2014-05-02}, 2014-06-25}, 2014-09-25}, 2014-12-25}, 2015-03-25}, 2015-05-04}, 2015-06-25}, 2015-09-25}, 2015-12-28}, 2016-03-25}, 2016-06-25}, 2016-08-28}, 2016-09-09}, VOO, 2016-09-25}, 2016-12-25}, 2017-02-02}, 2017-02-03}, 2017-02-14}, XAUUSD, VGLT, 2017-03-25}, VBR, 2017-04-06}, 2017-04-07}, 2017-04-19}, 2017-06-25}, 2017-07-05}, 2017-07-06}, 2017-07-13}, 2017-09-25}, 2017-10-05}, 2017-10-13}, 2017-12-27}, 2018-01-10}, 2018-01-29}, 2018-03-08}, 2018-03-12}, 2018-03-26}, 2018-04-03}, 2018-05-16}, 2018-06-25}, 2018-07-03}, 2018-09-26}, 2018-10-15}, 2018-11-26}, 2018-12-26}, 2018-12-27}, 2019-01-03}, 2019-01-28}, 2019-03-25}, 2019-03-29}, 2019-06-06}, 2019-06-25}, 2019-07-03}, 2019-07-08}, 2019-09-25}, 2019-10-07}, 2019-10-17}, 2019-11-27}, 2019-12-25}, 2020-01-06}, 2020-01-08}, 2020-01-29}, 2020-03-17}, 2020-03-25}, 2020-04-02}, 2020-04-03}, 2020-06-25}, 2020-07-21}, 2020-09-25}, 2020-10-16}, 2020-12-25}, 2020-12-30}, 2021-01-07}, 2021-01-25}, 2021-02-25}, 2021-03-15}, 2021-03-25}, 2021-04-25}, 2021-04-26}, 2021-05-07}, 2021-05-12}, 2021-05-13}, 2021-05-25}, 2021-06-01}, 2021-06-25}, 2021-06-30}, 2021-07-25}, 2021-08-25}, 2021-08-31}, 2021-09-25}, 2021-10-07}, 2021-10-08}, 2021-10-25}, 2021-11-08}, 2021-11-25}, 2021-12-25}, 2022-01-03}, 2022-01-25}, 2022-02-25}, 2022-03-04}, 2022-03-07}, 2022-03-25}, 2022-04-04}, 2022-04-25}, 2022-05-25}, 2022-06-06}, 2022-06-27}, 2022-07-11}, 2022-07-25}, 2022-08-25}, 2022-09-25}, 2022-10-03}, 2022-10-25}, 2022-11-14}, 2022-11-25}, 2022-12-27}, 2023-01-11}, 2023-01-25}, 2023-02-25}, 2023-03-06}, 2023-04-11}
Warning: Non-standard indentation.
Changes stay in memory until you [w]rite them out.
Type `h` or `?` for a list of available commands.
The b option does nothing with currency signs. It only loads account names, as reported by Beancount. This looks like it's related to the structure of your journal. You seem to have many transactions with implied currency signs and some with additional metadata, which bean-add is not designed to deal with. I'll add this to my list of things to consider.