crawlab
crawlab copied to clipboard
Allow for installing a list of dependencies
Would it be possible to add an option to install a list of dependencies? Eg. in the form of a requirements.txt file or similar? When adding several dependencies I have to search for and install one at a time.
It would also be nice to be able to install dependencies via command line which will also speed up installation process.
Thats really great, I was going to open a request for it. I was thinking of also retaining the installed/required dependencies in the database and when every new worker joins it would auto install all dependencies
It would also be nice to be able to install dependencies via command line which will also speed up installation process.
@username-mike and @saleh-hom basically you can do it in two ways.
-
Press
Play
button in your spider and typepip install <lib1> <lib2> <lib3>
to command input and chooseMode
->All Nodes
and after pressConfirm
btn. -
Or you can create
requirements.txt
with your deps in one of your spiders and run script with the following command for all Nodes:pip install -r requirements.txt