ArcticDB icon indicating copy to clipboard operation
ArcticDB copied to clipboard

Automated nightly release workflow

Open IvoDD opened this issue 10 months ago • 2 comments

Is your feature request related to a problem? Please describe. Currently releases are painful because:

  1. We occasionally discover issues/bugs during or after a release which is bad
  2. The full iteration of doing a release is slow even if there are no issues (build times, conda forge prs etc.)
  3. We rarely do releases and power users need to wait a long time to get new features

Describe the solution you'd like Having a separate automated nightly release process will have the following benefits:

  1. It will serve as a canary release and signal potential issues prior to a regular release (including issues with the release process itself).
  2. We can use the nightly build artifacts to quickly promote them to a full release.
  3. Power users will have access to new features sooner inside the nightly release.

This could look like an automated workflow which will run every night and push a new version to a separate python package e.g. arcticdb-nightly. Since arcticdb-nightly will not have a strong requirement for being bug-free the workflow can be fully automated without any human intervention.

We can then eventually transition internal Man Group use of arcticdb to the nightly build (and maybe allow for a flag to switch over to the latest regular release in case there are issues with the nightly build).

IvoDD avatar Apr 16 '24 09:04 IvoDD

We could probably do this by publishing a new RC every night or two, rather than a whole separate package.

poodlewars avatar Apr 18 '24 10:04 poodlewars

Nightly releases definitely can help fixing bugs and regressions as soon as possible.

I also do not think a separate package is needed, a separate index or channel can be used generally so that one can install the nightly releases with:

python -m pip install --pre --upgrade --extra-index-url <channel_for_nightly_builds> arcticdb

Projects in the PyData / Scientific Python ecosystem have been discussing publishing nightly releases and eventually converged on an agreement and a set of tools to automate them which might be helpful for ArcticDB.

jjerphan avatar Apr 18 '24 11:04 jjerphan