simonmichael
TWR calculation ignores prices on the day of cashflow operation
$ cat /tmp/test.journal
2024-06-18 buy A
assets:investment 1A @@ 100
assets:cash
;P 2025-06-17 A 110
2025-06-18 sell A
assets:investment -1A @@ 110
assets:cash
$ hledger -f /tmp/test.journal roi --inv investment --pnl "income expenses" --value=then, --infer-market-prices
+---++------------+------------++---------------+----------+-------------+-----++--------++------------+----------+
| || Begin | End || Value (begin) | Cashflow | Value (end) | PnL || IRR || TWR/period | TWR/year |
+===++============+============++===============+==========+=============+=====++========++============+==========+
| 1 || 2024-06-18 | 2025-06-18 || 0 | -10 | 0 | 10 || 10.00% || 0.00% | 0.00% |
+---++------------+------------++---------------+----------+-------------+-----++--------++------------+----------+
$ hledger -f <(cat /tmp/test.journal | tr -d ';') roi --inv investment --pnl "income expenses" --value=then, --infer-market-prices
+---++------------+------------++---------------+----------+-------------+-----++--------++------------+----------+
| || Begin | End || Value (begin) | Cashflow | Value (end) | PnL || IRR || TWR/period | TWR/year |
+===++============+============++===============+==========+=============+=====++========++============+==========+
| 1 || 2024-06-18 | 2025-06-18 || 0 | -10 | 0 | 10 || 10.00% || 10.00% | 10.00% |
+---++------------+------------++---------------+----------+-------------+-----++--------++------------+----------+
Now it is indeed the "intraday valuations are hard" issue. TWR calculation splits everything into periods and the period ends one day before the actual transaction so that "--infer-market-prices" rate is not applied to that period. Note -- the P directive is dated one day before the "sell" transaction, so that TWR could see it. If the date was the same day TWR would also ignore it.
The issue becomes less drastic if you do have a P directive for every date assuming the price change over one day is not as great as over the actual period. Still the TWR should have a subperiod of "just before" the actual transaction and without real intraday valuations this isn't going to happen.
I can only add that I can reproduce this, and running the commands with -b 2024 -e 2027 --cashflow --debug 3 and diffing the output shows that pricing functions do end up using different prices in these two cases:
@@ -546,11 +546,11 @@
default valuation commodity for A: Just ""
seeking A to price using forward prices
forward prices:
- 2024-06-18 A> 100
+ 2025-06-17 A> 110
trying: A>
shortest path from A to : A>
price chain:
- 2024-06-18 A> 100
+ 2025-06-17 A> 110
valuation date: 2025-06-18
default valuation commodity for A: Just ""
seeking A to price using forward prices
I chose a long reporting inteval to show that inteval splits do not contribute to the outcome.
The TWR computation needs to compute the value of investment prior to the transaction on 2025-06-18. As we cannot compute intraday values, the prior is computed as the value at the end of 2025-06-17.
Without the P directive, it is computed to be 100. With P directive, it is 110.
I think the argument that @aragaer is making is that --infer-market-prices should lead to the pre-transaction value of the investment to be computed using the inferred price, but I think it is equally easy to argue the opposite approach.
I think this is impossible to fix in a robust way unless we have some variant of the global ordering of transactions, and ability to compute the balance after arbitrary transaction is this ordering.
Is it correct to generalise this issue title as "TWR calculation ignores prices on the last day of report period" ?
Most reports treat the report end date as exclusive, but I think close might be one that treats it as inclusive, for ergonomic reasons.
Until/unless some better change arises, what advice could be added to the doc so people won't be surprised and can easily work around this for practical reporting ?
An unrelated question: why does it report cashflow as -10 there, and not plus 10 ?
PS: what levels of the affects- and annoyance- labels do you think best fit this issue ? (Mouse over the label values to see descriptions). Feel free to add them yourself if you have access.
An unrelated question: why does it report cashflow as -10 there, and not plus 10 ?
Cashflows that increase the value are reported as positive, cashflows that decrease the value as negative, so it is essentially change-of-value from the perspective of the investment account
Is it correct to generalise this issue title as "TWR calculation ignores prices on the last day of report period" ?
I would disagree with this. My attempt would be: "it is hard to compute the current balance/value of the account right before a particular transaction in a way that would work in the most general case". The most general case being "there are multiple transactions on the same date, and they all use different commodity prices, and we want to be able to compute the value before each and every one of those transactions". (yeah, that's a handful :)
In this particular case there is just one transaction on any given date, but the core of the issue is the same. If we want to get the value (or the price, which is the same thing) prior to the transaction on date T, we currently have to take the price from T-1 - any other approach that uses prices from T exposes you to price/value after the transaction.
I would even go as far as to claim that ability to compute the value immediately before (or after) the transaction (in the presence of inferred prices) is the cornerstone the TWR computation -- and I am not talking about roi.hs, I am talking purely about the algorithmic task of computing TWR whichever way you like (pen and paper, excel,...)
Most if not all of all TWR-related tickets so far come back to this specific requirement again and again, and I suppose an argument could be made that it might be better to just remove the TWR code (because it works for simple journals, but starts to come apart at the seams when users get creative or make reasonably-looking assumptions that you just can't express on the journal syntax and therefore can't robustly support)
It isn't just about periods -- in cashflow calculations each cashflow event counts as a new period.
Looking at the wikipedia article I see that there's another way to calculate TWR -- "right after the cashflow event". I believe that approach might result in numbers closer to what's expected even though it won't be "exactly after".
It would solve this specific use-case, but 'break" others :)
For instance, if you manage to draw down the entirety of your investment multiple time Sutton the reporting period, you end up with several points where your "value after" is zero, and that creates an inconvenient corner case that you need to handle specially
A couple of versions ago TWR used "valuate after" approach, and it was worse than what we have right now IMO
"it is hard to compute the current balance/value of the account right before a particular transaction in a way that would work in the most general case"
But for a title that communicates exactly what a user will see because of this issue: is "TWR calculation ignores prices on the last day of report period" (both prices declared by P, and prices inferred from costs) correct ?
More or less yes. This "last day of period report" is not really a part of the period. It's the first day after the period (and the first day of the next period). And period in TWR calculation is actually a "period between cashflows". So the title really is "TWR calculation ignores prices on the day of cashflow operation".