chronon
chronon copied to clipboard
[CI] Set a time limit for circle ci
Summary
Setting a time limit of 40 minutes for circle CI as metadata calls can run into a 4hr+ loop of failure Example: https://app.circleci.com/pipelines/github/airbnb/chronon/3408/workflows/2fdc3dd8-904b-4d90-b062-d5879f4bd686/jobs/17449
From: https://support.circleci.com/hc/en-us/articles/11775183948827-How-to-run-a-job-in-the-background-on-CircleCI
A job that has background: true set will run while there are other jobs running but once they have finished the background job will also end, it will not run on its own if there are no other jobs.
Pick 40 minutes since both Spark 3.1.1 and 3.2 run in ~10-20minutes. Spark 2 can run for 40 minutes but rarely will provide more info than the other two together.
Why / Goal
Improve dev experience.
Test Plan
- [ ] Added Unit Tests
- [x] Covered by existing CI
- [ ] Integration tested
Checklist
- [ ] Documentation update
Reviewers
anyone.