apm-server icon indicating copy to clipboard operation
apm-server copied to clipboard

[META] Benchmark 2.0: Write benchmark intake event data upload tool

Open marclop opened this issue 3 years ago • 1 comments

Description

We would like to stop committing the generated event data recorded by the intake-receiver that is consumed by apmbench to the main apm-server repository. Currently the events are stored in systemtest/benchtest/eventsfor convenience, but it is not ideal to commit such large files to the git repository, particularly as we add more datasets from different agents and evolve the generated data set.

The main goal of this tool is to allow uploading and downloading the events from where they're stored (most likely we should look into leveraging blob storage services from cloud providers) and perhaps filter which files are downloaded, for example, allow downloading only a single language agent's benchmark (such as go-2.0.0.ndjson), and which version of the agent should be downloaded (latest or a specific revision).

We should also explore if and how we'd like to some versioning on the files that are uploaded.

marclop avatar Apr 11 '22 12:04 marclop

Let's aim for a very simple first version of such an uploader for 8.3 (not considering it highest priority for the 8.3 work).

simitt avatar Apr 19 '22 18:04 simitt