piker icon indicating copy to clipboard operation
piker copied to clipboard

Better `TOML` libs -> `tomlkit` for style, `tomli(-w)` for speed

Open goodboy opened this issue 2 years ago • 3 comments

Since we already had to roll our own based on writing a particular styled .clears table to the pps.toml file, this is a task to choose the best (combo of) lib(s).

Here's the verbatim bullet-task breakout from #345:

  • after much pain and anguish getting the toml encoder to output things sanely, i think it's just worth writing or switching to a super fast custom encoder that writes adhoc-ly in the format we want for positions entries in pps.toml, the reader-writer separation is already what tomli does (the fastest pure py reader soon to land in the stdlib) and then we can just drop toml outright and worry about getting a general encoder later.

  • [x] support better indentation per broker-account for better readability, indented tables per broker-account would be super nice if possible

    • see tomlkit below which supports this -> https://github.com/pikers/tomlkit/blob/writing_docs_tweaks/tomlkit/items.py#L1644
    • see our PR adding an indent control to Array.multiline() https://github.com/sdispater/tomlkit/pull/294

Turns out there's been a bunch of project updates in the space 🙏🏼,

  • tomlkit: from the poetry/pendulum auth: https://github.com/sdispater/tomlkit and is based on a rust lib :godmode:

    • [x] contains multi-line array writing: https://github.com/pikers/tomlkit/blob/writing_docs_tweaks/tomlkit/items.py#L1122
    • [x] supports inline table rendering: https://github.com/pikers/tomlkit/blob/writing_docs_tweaks/tomlkit/items.py#L1122
    • [x] we can now work off our own fork: https://github.com/pikers/tomlkit
      • [ ] needs Decimal input support: https://github.com/sdispater/tomlkit/issues/288
      • [ ] needs bidict input support: https://github.com/sdispater/tomlkit/issues/289
  • msgspec.toml now exists :party: which means we can natively render our msgs and structs easily!

    • uses tomli(-w) underneath so using that probably makes the most sense for our speed solution.
    • https://jcristharif.com/msgspec/install.html#toml
    • https://jcristharif.com/msgspec/api.html#toml
    • https://jcristharif.com/msgspec/supported-types.html#datetime
  • a list of other alt libs that we prolly won't use but figured might as well put them here just in case:

    • pytomlpp c++ wrapped parser: https://bobfang1992.github.io/pytomlpp/pytomlpp.html
    • toml-w, stdlib's toml sibling writer lib: https://github.com/hukkin/tomli-w#does-tomli-w-support-writing-documents-with-comments-or-custom-whitespace
      • [ ] has Decimal support which i don't think is in tomlkit: https://github.com/hukkin/tomli#construct-decimaldecimals-from-toml-floats

goodboy avatar Mar 24 '23 21:03 goodboy

personal todo: Need to link our PR for inline array comment config support and possibly open an issue to support passing through such inputs to .item.array()

goodboy avatar Apr 29 '23 00:04 goodboy

Linking in tomlkit issues we need in order for us to be able to use that lib:

  • https://github.com/sdispater/tomlkit/issues/288
  • https://github.com/sdispater/tomlkit/issues/289
  • https://github.com/sdispater/tomlkit/issues/290

goodboy avatar May 13 '23 20:05 goodboy

Heh, i actually was on the side of this issue / request submitted to the spec: https://github.com/toml-lang/toml/issues/516

After reading it though, i don't think multi-line tables really are that necessary.

Interested in what others think though as well.

goodboy avatar Jun 19 '23 15:06 goodboy