recipy
recipy copied to clipboard
recipy
Fancier way of dealing with conda deps for testing
In #207, we want to only install keras if we're on certain versions of Python. That means we can't easily include it in the first call to conda install - we have to have an if statement testing the version and then installing it. However, this means that conda might decide to silently downgrade Python when we do the install. For example, the Travis test config says to run:
conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION
pytest
python-dateutil pyyaml numpy git
matplotlib
This creates an environment where everything is compatible with everything else. Great. But if we then add something, inside an if statement for different Python versions, that does:
conda install keras
We might find that command actually downgrades Python from $TRAVIS_PYTHON_VERSION to a lower version (or even upgrades). This would result in us testing on a different version to the one we thought we were testing on. It would also take quite a while, as conda might install a load of different versions of packages once it realises the keras needs to be installed - so they should really be installed at the same time in one go.
One potential way to solve this would be to have a separate requirements.txt-style file for each Python version, and to install from this. Unfortunately this can't be a conda environment.yml file, as that encodes the actual versions of each dependency, and we want to keep up-to-date with the latest versions that can be installed automatically by conda.
Therefore, I propose we create a Python script called set_up_conda_environment.py which is called in the testing script. It checks the Python version, then grabs the appropriate file - for example conda_requirements_3.7.txt which contains a list of packages, separated by newlines. This is then converted to a space-separated list, which is put together into a conda create call, and executed via os.system (actually subprocess.check_output or something).
So, for example, we could replace this in the current .travis.yml:
conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION
pytest
python-dateutil pyyaml numpy git
matplotlib
with this:
python set_up_conda_environment.py $TRAVIS_PYTHON_VERSION
and a file called conda_requirements_3.7.txt containing:
pytest
python-dateutil
pyyaml
numpy
git
matplotlib
This file could be different for each version, to install the relevant requirements for each version (or platform, if necessary).
This would probably be a change that would be required before #207 could be fully tested and merged - so what do you think about me going ahead with this @jvdzwaan?
It sounds like a bit of a hassle, also to maintain, but I don't have suggestions for a better (simpler) way.
It is probably a good idea to install everything using a script and not have some things in .travis.yml and others in the script (but I guess that is exactly what you are suggesting).
Would there be a way to mark things that should be pip installed? Keeping track of one requirements file for each python version is a hassle, but having to keep track of two files with requirements (one for conda and the other for pip) would be worse.
it should be easily contained within one python script for each python version.
