deepforge
deepforge copied to clipboard
Default installed packages in workers
Although operations can define their own custom dependencies, it might be nice to include more default packages in the operations. This is especially motivating since conda is somewhat slow so installation of these common data science packages may be a bit annoying.
@umesh-timalsina and I have been talking about this somewhat recently and @dustinjoe also brought it up recently here so it is probably about time it got an official issue :)
If we decide to go this route, some candidates for being included in the base environment:
- scikit learn
- numpy
- pandas
This would be nice. One of the issues I faced while working on #1747 is that the more packages there are in your base environments, the more it takes for conda to export it. It will not be an issue for local
and gme
computes, but can cause lag problems in for example -> ephemeral computes like sciserver-compute
, where base conda environments do not exist