OTAnalytics icon indicating copy to clipboard operation
OTAnalytics copied to clipboard

Op#3648 create streaming track parser

Open KueblerJelle opened this issue 5 months ago • 0 comments

Introduced streaming track and detection parser.

Detections in json format are parsed as streams and grouped to tracks. The finished flag marks the end of a track. After parsing the last detection of a track, that track is provided to the stream of tracks.

Added new implementation of Cli and added command line arguments: cli_mode (stream or bulk), cli_chunk_size (in case of streaming mode, this confidures the number of tracks to be collected before performing cuting/counting etc.)

To reduce the memory consumption, track and and event repository should be cleared regularly. This however requires the export of results to work incrementally. Therefore an export mode is introduced:

  • Overwrite -> write the given results into the file, overwrite existing results
  • Initial Merge -> write or collect results incrementally, on first merge maybe include metadata such as csv header
  • Merge -> subsequent incremental results (append or collect)
  • Flush -> final result increment, if results were collected in memory, flush/write to file

Adapted all current result export implementations to respect the export mode.

Parameterized test_cli and regression_otanalytics to test both cli_modes.

KueblerJelle avatar Sep 18 '24 07:09 KueblerJelle