structure icon indicating copy to clipboard operation
structure copied to clipboard

Increase "entities update since" performance

Open neopostmodern opened this issue 10 months ago • 0 comments

All stats for roughly 8.5k notes and 3k tags

General

  • leanTypeEnumFixer has <10ms impact
  • The majority of DB-related time (>90%) is spent pulling data, not querying/getting a cursor. This was tested by running the query in mongosh with .explain("executionStats") and .toArray().

First read

  • The majority of the time is now spend outside entitesUpdatedSince (<2s out of 7–8s)
  • Notes DB read is 1–1.5s, tags 0.2–0.3s

Subsequent reads

  • For subsequent calls the majority of the time is spent pulling note and tag data from the DB (60–70% and 20–30% of 0.8–1s respectively).
  • The time to build the diffs is minimal (1–3% combined) – but only if the diffs are empty!
    • If there is any difference, but it is small, the diffing time jumps by ~100ms (10% of total time extra).
    • Full chaotic diffs (e.g. from forgetting to sort consistently) results in 5+ minutes of diffing.
    • There is a consistent overhead around the actual (library provided) diffing of 1–3ms.
  • Cache operations are not very time-intense (<5%)
    • empty diff: read 1–3%, preprocessing <1%, write <1%
    • minimal diff: read 1–3%, pre-processing 1–2%, write 1–2%
  • ... (document other changes and learnings)

State pre-optimization (master)

Pristine read: 35–36s Empty diff subsequent read: 0.8–1s Minimal diff subsequent read: 1–1.2s

Stats "after optimization"

Pristine read: 7–8s Empty diff subsequent read: 0.7–1s Minimal diff subsequent read: 1.1–1.2s

neopostmodern avatar Apr 20 '24 23:04 neopostmodern