re-frame-10x icon indicating copy to clipboard operation
re-frame-10x copied to clipboard

Rebuild Epoch Parsing

Open superstructor opened this issue 3 years ago • 0 comments

The epoch parsing code is in need of significant refactoring and bug fixes. It has led to the following issues:

  • delays in rendering content: #264
  • lagging event history: #268
  • dispatch-sync invisible: #154
  • nil app-db state: #263, #151
  • inconsistent use of event filters: #175
  • duplication of subs between the current epoch and intra-epoch subs
  • invalid timing information

The relevant code is in ::receive-new-traces, parse-traces, surrounding code in the metamorphic ns and in subs that use the app-db state derived from the epoch parsing.

Step 1: Move ignored-events filtering from event to subs

In ::recieve-new-traces there is a remove to filter ignored events (via settings) before epochs are stored in app-db.

This causes some issues:

  1. Partial epoch states in app-db leading to odd workarounds in subscriptions which are not all handled
  2. If the user removes or adds an epoch filter with existing traces in app-db, 10x is in an invalid state as it does not have a correct 'view' of epochs according to the settings.

The remove filter needs to be deleted from ::recieve-new-traces and added to the appropriate subs instead.

Step 2: Change metamorphic/parse-traces so that dispatch-sync is its own epoch

The delimitation of dispatch-sync epochs is not detected in metamorphic/parse-traces. This needs to be fixed so a dispatch-sync is returned as a distinct epoch.

Step 3: Move all parsing from ::recieve-new-traces event to metamorphic ns

::receive-new-traces contains too much data transformation/parsing related code. If this is moved to metamorphic ns it would be easier to write unit tests to debug/check correct behavior and so forth.

New APIs/functions should be considered as part of this move, as should renaming. The current names are certainly not great.

Tests should be added for all the significant epoch parsing use cases/situations.

At least the following should be moved:

Step 4: Performance

Epoch parsing and associated code is currently the biggest hit on 10x performance.

Once more of the code is moved into functions in metamorphic with unit tests various optimisations could be explored such as storing the internal parse state as transient vs persistent collections, doing less string parsing etc.

superstructor avatar May 25 '21 05:05 superstructor