Run lakeFS load tests on a each PR - Discovery

[talSofer]

Use https://github.com/treeverse/lakeFS/tree/master/cmd/lakefs-loadtest to run load tests on a recurring cadence, for main lakeFS workflows.

[ortz]

Taking this one as we have a similar item following an incident we had.

[ortz]

I tried running the load tests using docker services which lets you run another container in the background and run commands (postgres in our case). In terms of performance, it takes a lot of time and doesn't seem feasible to hold the load tests for lakeFS. I think these load tests should run against a real environment (staging?) with the PR-related built artifact, which requires proper discovery and planning.

I'm writing down how we should test it manually in the meantime.

1. On lakeFS PR creation, we should create a docker image (as the last step)
2. Deploy that docker image into staging environment (a dedicated isolake or existing one)
3. Run load tests (using the lakefs-loadtest command)
4. Revert the isolake version
5. Save the results in a database
6. Display the benchmark results as part of the PR
7. Fail the test if there's a big change in the test run compared to the average benchmark in the last 3 runs.

Changing the scope of this ticket to discovery. @guy-har @idanovo @eladlachmi fyi

[github-actions[bot]]

This issue is now marked as stale after 90 days of inactivity, and will be closed soon. To keep it, mark it with the "no stale" label.

[github-actions[bot]]

Closing this issue because it has been stale for 7 days with no activity.