gds_env icon indicating copy to clipboard operation
gds_env copied to clipboard

Explore using `conda-lock` to generate lockfiles

Open darribas opened this issue 2 years ago • 2 comments

conda-lock provides the ability to generate lockfiles for different platforms. This would be a great addition to replace the currently gds_py.yml as a way to build the same Python environment as the container natively.

An idea of how this could work:

  • Build gds_py environment on Dockerfile
  • Export environment into one of the supported input files
  • Generate lockfiles for Linux (Arch/arm), macOS (intel/M1), Windows
  • Publish lockfiles with each release so the environment is fully recreateable natively on a broader range of machines without conflicts

If the workflow above can work, then we should add CI testing on each architecture.

darribas avatar Apr 13 '22 19:04 darribas

An alternative to conda-lock that might work just as well through Github Actions would rely on vanilla conda ability to build identical environments:

  • Build gds_py on the Dockerfile
  • Export environment to gds_py.yml as now
  • Upon commit to this repo, a Github Action would:
    • Trigger the build of the environment on a variety of supported architecture/OS/platform options
    • Test the environment passes the tests
    • Write out a full spec file with:
    > conda list --explicit > gds_py-<architecture>-<os>-<platform>.txt
    
    • Commit the spec files to the repository

These spec files would later be able to be used directly to replicate the environment in different machines:

> conda create --name gds --file gds_py-<architecture>-<os>-<platform>.txt

One major drawback of this approach is the end-user needs to know their architecture/OS/platform to select the appropriate install file.

darribas avatar Apr 13 '22 20:04 darribas

Arch's to be supported (and write explicit files for):

  • win-64
  • osx-64
  • osx-arm64
  • linux-64
  • linux-aarch64

darribas avatar May 13 '22 09:05 darribas