--infer-market-prices is not limited by queries

[chrpinedo]

I am not sure if this is a bug or if it is something to live with. I only open the issue for your consideration.

Description: when --infer-market-prices is used all the transactions and postings are considered for pricing purpose, even if those are excluded in the current search.

Example:

Journal file:

2022-10-01 begin
    assets:checkings  100.00 EUR
    equity:open/close balance

2022-10-01 buy stocks
    assets:stocks  10 AAA @ 10.00 EUR
    assets:checkings

P 2022-11-01 AAA 20.00 EUR

2022-12-31 close books  ; clopen=2023
    assets:stocks  -10 AAA @@ 100.00 EUR
    equity:open/close balance

2023-01-01 open books  ; clopen=2023
    assets:stocks  10 AAA @@ 100.00 EUR
    equity:open/close balance

When the value is calculated without --infer-market-prices, I get the good value:

$ hledger -f tmp.journal bal -V                                          
          200.00 EUR  assets:stocks
         -100.00 EUR  equity:open/close balance
--------------------
          100.00 EUR

When the value is inferred I get bad value:

$ hledger -f tmp.journal bal -X EUR  --infer-market-prices               
          100.00 EUR  assets:stocks
         -100.00 EUR  equity:open/close balance
--------------------
                   0  

Even If I exclude the clopen entries:

$ hledger -f tmp.journal bal -X EUR  --infer-market-prices not:tag:clopen
          100.00 EUR  assets:stocks
         -100.00 EUR  equity:open/close balance
--------------------
                   0  

This behavior can be a bit confusing, specially when we close books. In the close books documentation it is proposed to use the clopen tag to exclude the open/close transactions when we want an inter-annual report. Well, if we use --infer-market-prices, the inferred price of clopen transactions is not excluded. This could be solved by different ways:

• change the --infer-market-prices behavior
• document that in case of using --infer-market-prices in the reports it is recommend to provide "P" directives to the open/close dates to avoid issues with the valuation of assets in inter-annual reports.

Version:

$ hledger --version                                                      
hledger 1.31, linux-x86_64

[wgslr]

Hi! I think the current behavior is correct, and in my use case it is more useful. I don't track stocks but have expenses and accounts in multiple currencies. And before I started using P directives, I was relying on infer-market-prices to get rough currency conversions. So even if all transactions today were in EUR, I could do hledger print -p today -X USD --infer-market--prices and get the prices in USD based on some older exchange transaction. With your approach, --infer-market-prices would become useless whenever a short time period was selected.

I see you point that it can be surprising, depending on your mental model, an inconvenient with the book closing transaction. I wouldn't call it a bug, but rather matter of ensuring the documentation is clear and perhaps creating a separate feature for "filtered" inferred prices

[simonmichael]

Thanks for the report and response. I tend to agree with @wgslr. I'll try to justify it as well: market prices are a kind of immutable background data, with a fixed value on each date. We normally read them from explicit P directives; or we can automate that more by generating the P directives from transactions with --infer-market-prices. Other than choosing valuation date, we don't get to/want to choose subsets of market prices. (If you did need that, you could arrange to put them in different price files and select the one you want at report time.)

[simonmichael]

Perhaps saying something about this at https://hledger.org/dev/hledger.html#--infer-market-prices-market-prices-from-transactions would be helpful ? A draft or PR is welcome

[chrpinedo]

I agree that perhaps it is not a bug, only a handicap of using --infer-market-prices.

But I found so disturbing the closing books case. Because the closing book documentation proposes to use the clopen tag to open and close the books. It is always possible to do inter-anual reports with not:tag:clopen, it is like not having closed the books. Well this is correct but only if it is not used --infer-market-prices because in that case values and hledger roi command provides wrong values for the dates of opening/closing transactions. Because although the opening/closing transactions are not taken into account for the current report, the inferred prices of those transactions are taken into account.

The solution for closing books and using the --infer-market-prices is to provide additional P directives for the books' opening and closing dates.

I would add some comment to https://hledger.org/dev/hledger.html#example-excluding-closingopening-transactions or I would close this issue. If anybody else faces again this issue, it can find the solution in this closed issue.

[simonmichael]

Thanks for the suggestion. I haven't really understood exactly where the problem arises with close. What comment would you suggest ?

The solution for closing books and using the --infer-market-prices is to provide additional P directives for the books' opening and closing dates.

Yes, that is (should be) our general recommendation when inferred market prices aren't working out ("...value reports can sometimes be affected in confusing/undesired ways by your journal entries...") - declare market prices explicitly with P.