astro-sdk
astro-sdk copied to clipboard
astronomer
Set the CI to run and publish benchmark reports once a week
Dependencies
- Depends on: #432
- Depends on: #434
- Depends on: #442
- Depends on one of:
- #438
- #431
- #437
Acceptance criteria
- [ ] The CI should run the benchmark & analysis scripts on a weekly basis for all scenarios defined
- [ ] If any of the benchmark scenarios had a performance degradation of 10% (took 10% of the time more than the previous execution of that same scenario) - we should notify in slack
- [ ] The markdown created by the analysis script should be committed to the repo
Just giving additional information, this feature was requested by @vikramkoka during a meeting with him - and the goal is to make sure our benchmark results are always up-to-date with the current performance of the Astro Python SDK.
@pankajastro explored Github Actions and will try out Github action along the lines of this ticket today.
There is a scheduled event in GitHub workflow but looks like it runs on the default branch only i.e main our case so we might have to merge PR in order to full end-to-end test. I have tested the scheduling part on my private repo it seems to be working. Currently, trying to test the benchmark make command in separate job once this succeeds then we should be able to combine both together
trying to run below job
Run-Benchmark:
runs-on: ubuntu-18.04
env:
GOOGLE_APPLICATION_CREDENTIALS: /tmp/google_credentials.json
GOOGLE_APPLICATION_CREDENTIALS_JSON: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_JSON }}
steps:
- uses: actions/checkout@v2
- run: python -c 'import os; print(os.getenv("GOOGLE_APPLICATION_CREDENTIALS_JSON", "").strip())' > ${{ env.GOOGLE_APPLICATION_CREDENTIALS }}
- run: cd tests/benchmark && make
Error: Error when reading or editing Project Service --****/container.googleapis.com: Failed to list enabled services for project astronomer-dag-authoring: googleapi: Error 403: Permission denied to list services for consumer container
trying to run below job
Run-Benchmark: runs-on: ubuntu-18.04 env: GOOGLE_APPLICATION_CREDENTIALS: /tmp/google_credentials.json GOOGLE_APPLICATION_CREDENTIALS_JSON: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS_JSON }} steps: - uses: actions/checkout@v2 - run: python -c 'import os; print(os.getenv("GOOGLE_APPLICATION_CREDENTIALS_JSON", "").strip())' > ${{ env.GOOGLE_APPLICATION_CREDENTIALS }} - run: cd tests/benchmark && makeError: Error when reading or editing Project Service --****/container.googleapis.com: Failed to list enabled services for project astronomer-dag-authoring: googleapi: Error 403: Permission denied to list services for consumer container
this permission has been resolved
Script failing while pushing image into registry
unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Makefile:37: recipe for target 'container' failed
make: *** [container] Error 1
Error: Process completed with exit code 2.
https://github.com/astronomer/astro-sdk/actions/runs/3067653209/jobs/4954955710
Script failing while pushing image into registry
unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication Makefile:37: recipe for target 'container' failed make: *** [container] Error 1 Error: Process completed with exit code 2.https://github.com/astronomer/astro-sdk/actions/runs/3067653209/jobs/4954955710
This got fixed after adding below step
gcloud auth activate-service-account --key-file $(GOOGLE_APPLICATION_CREDENTIALS)
Benchmark script succeeded in CI https://github.com/astronomer/astro-sdk/actions/runs/3092321371/jobs/5003571780 commit: f3d0a7c672d8eac990fa8f1d00b136ad0f9b5663
