Position accounting refinements, bug fixes, src-asset-pnls, `ledgerd` daemon

[goodboy]

The major summary follow up to the .accounting subsys introduced in #489 this is the list of outstanding but needed features and bugfixes that should be resolved asap!

The existing #478 was the older not-as-specific issue for this stuff..

The new set of top level issue-tasks are:

• [ ] new toml libs: #496
• [ ] accounting schema changes: #510
• [ ] ledger CLI and offline audit system: #512
• [ ] paper engine clearing models: #476
• [ ] dark and alert testing: #463
• [ ] dark / alert offline persistence: #463
• [ ] order mode UI outstanding bugs: #479
  • [ ] #267
  • [ ] #248
• [ ] missing order mode features:
  • [ ] #224
  • [ ] #220

The high level sections here are:

• missing accounting calcs and documentation

  • (eg. ppu, bep)
  • cross asset pnl valuations

• UI components to display the above ^

• service daemon ledgerd or accountd to do bookkeeping out of thread

---

Missing critical calcs and accounting:

• [ ] as per #510, make Position.clears -> Position.events as per #510 to handle market specific accounting calc modifying occurrences (like stock splits)!

• [ ] use transfers/withdrawals to modify the Position.ppu

  • particularly important for cexes such that ppus are calcuated properly and graphically correct on chart!

• [ ] add a Position.bep:

  • [ ] TODO: white board math is as follows:

Pertaining notes and tasks from legacy issues:

• #263, the original kraken position tracking way back before we had anything :joy:

  • how spot is junko compared to their margin trading..
    • they do offer pp tracking on margin trading positions but not spot?
    • https://support.kraken.com/hc/en-us/articles/203325683-Differences-between-spot-exchange-and-margin-trading
  • Things that are either confusing af or of note in their api:
    • they have "trade balance" endpoints but only for margin: https://docs.kraken.com/rest/#operation/getTradeBalance
      • also wth don't they explain what all the result keys are 😂
    • here's where you can pull past spot fills/trades, but they limit to 50: https://docs.kraken.com/rest/#operation/getTradeHistory

• #373, more kraken accounting todos from the comments:

  • [ ] for wallet transfers out of an account we get a Transaction-like entry in our account.<>.<>.toml of the form (note this is a non-real example):

    [xmr/eur.kraken]
    size = 78.31125778
    ppu = 0.3222278400000044
    bsuid = "xmr"
    clears = [
     { dt = "2022-12-21T07:20:16.316061+00:00", ppu = 0.3222278400000044, accum_size = 778.33125778, price = 0.3212, size = 778.33125778, cost = 0.4, tid = "TJKLRL-7UE2L-BX2UE5" },
     { dt = "2022-12-21T07:26:56+00:00", ppu = 0.3222278400000044, accum_size = 78.31125778, price = "NaN", size = -700.02, cost = 0, tid = "1503A92E24BAA1E8B8E7B78E79A78AAF3E6C3F3E35227EF7D2C13754876B37D5" },
    ]
    

    • ^ Notice the price field has a Nan value
    • [ ] this transaction record should contribute to the rolling ppu calc in the sense that on the next clear the weighting of the position should be more right?
      • well that would be the case where you presume the withdrawn funds take no losses when (eventually) used, which seems like the sanest way to copy with tracking this account's pp right?
      • what happens if you make further trades in this market-pair and then bring back the transferred out funds and begin to sell them?
    • [ ] how do we handle buying more of xmr from say another pair in terms of the current next ppu weighting in this position?
      • For eg. if you bought xmr/btc how do we project that contribution to this position?
        • say you wanted to project it back to some "main fiat" like EUR in this case?
        • the EUR value would come from the original (recursive) projection of btc/eur, which would provide the source/base pnl from preceding trades as they contribute to the projection to EUR; seems like the calculation needs to be recursive and rely heavily on time stamps?

  • [ ] we aren't tabulating nor using the widthdrawal fee schedule in our cost field in each ledger entry, obviously this should resolve any accounting discrepancies a transfer-as-sell clear should have a cost (that kraken takes) as part of the position calc.

---

UI stuff we'll probably be innovators of..

How about historical position line graphics on charts which show breakeven price changes overlayed with actual underlyings, aka pp liftetime "lines" more or less as super simple FSP style graphics that show the beginning, changes, and end to a given pp such that users can easily see past trades and their state changes on top of historical data :surfer:

• [ ] as per #220 and much idea-ing during #489
• [ ] bep and ppu historical lines over every chart with real-time update with flat lines.
• [ ] on order staging dotted-line project both the bep and ppu given the current level line value B)
  • idea is that for any order you're planning to make you see how it will change both your ppu and bep!
  • would change in rt based on user's cursor price placement
  • what to do about submitted live order?
    • some kinda line projection kept up to date like the ppu / bep lines?
• [ ] will likely require that we also compute and store arrays of breakeven or lifo prices per clear in the account<broker>.<account>.toml
• might be handy to show actual LIFO style "contextual" pricing if configured which could make it easy to enter contextual short term trades despite an overall breakeven price, egs:
  • enters long but bep-pnl is red but there's a dump and you want to play a bounce so you complete a bounce trade and make a profit and then your be price goes back to what it was before the scalp trade
  • same but say you have a green pnl (in this case entering above your be would bring it's mean price up) and want to show your original longer term be_price after the trade completes
• [ ] adding "ghost data" for paper engine fills which we can display as a curve overlay to the actual price history?
  • for example filling in (lesser colored) line/bar to show the theoretical paper engine clearing history for something like this:

---

Add a ledgerd actor-daemon?

Should we consider a separate actor-service: ledgerd which can do all ledger record monitoring (via connections to emsd) and writing out-of-process from brokerd/datad such that all other actors only deal with position data via msgs?

Currently the fs IO is directly serving latency for broker backends which call this code in thread during startup and possibly on clearing / updates.

Premises:

• it's way more handy to have real-time updates as a user, also obvsly for testing..

Solutions:

• [ ] a singleton-style actor that listens to msgs from the ems and does all the work of computing positions and recording them to whatever backing storage sys we have (for now account.broker.account.toml files) and thus can operate as the writer in a readers-writer style lock around file access.

• [x] we should add support for immediate config file (ledger or pps.toml) writing

  • [x] initial support for this will land in #489 via commit 2f31e40d3
  • [ ] maybe/eventually via the trio async file IO apis which use a bg thread to avoid main thread latency.

• [ ] can we make a better incremental update system;

  • right now we have lots of file writes going on in both the paper engine, per clear, and in various brokers. backends inside the trades_dialogue() endpoint.
  • using a timestamp scanning technique in TOML and/or a different filesystem format (like parquet? something binary?) we can probably accomplish this rather easily.
  • ~~only do ledger file writes on teardown/api-block exit by default~~

[goodboy]

Might be worth looking at the semantics used by some not-total-garbage brokers for comparison and inspiration as well,

• https://support.kraken.com/articles/portfolio-performance-on-kraken-pro