ArcticDB
ArcticDB copied to clipboard
Automated nightly release workflow
Is your feature request related to a problem? Please describe. Currently releases are painful because:
- We occasionally discover issues/bugs during or after a release which is bad
- The full iteration of doing a release is slow even if there are no issues (build times, conda forge prs etc.)
- We rarely do releases and power users need to wait a long time to get new features
Describe the solution you'd like Having a separate automated nightly release process will have the following benefits:
- It will serve as a canary release and signal potential issues prior to a regular release (including issues with the release process itself).
- We can use the nightly build artifacts to quickly promote them to a full release.
- Power users will have access to new features sooner inside the nightly release.
This could look like an automated workflow which will run every night and push a new version to a separate python package e.g. arcticdb-nightly
. Since arcticdb-nightly
will not have a strong requirement for being bug-free the workflow can be fully automated without any human intervention.
We can then eventually transition internal Man Group use of arcticdb to the nightly build (and maybe allow for a flag to switch over to the latest regular release in case there are issues with the nightly build).
We could probably do this by publishing a new RC every night or two, rather than a whole separate package.
Nightly releases definitely can help fixing bugs and regressions as soon as possible.
I also do not think a separate package is needed, a separate index or channel can be used generally so that one can install the nightly releases with:
python -m pip install --pre --upgrade --extra-index-url <channel_for_nightly_builds> arcticdb
Projects in the PyData / Scientific Python ecosystem have been discussing publishing nightly releases and eventually converged on an agreement and a set of tools to automate them which might be helpful for ArcticDB.