crawlab icon indicating copy to clipboard operation
crawlab copied to clipboard

Allow for installing a list of dependencies

Open oyhel opened this issue 1 year ago • 3 comments

Would it be possible to add an option to install a list of dependencies? Eg. in the form of a requirements.txt file or similar? When adding several dependencies I have to search for and install one at a time.

oyhel avatar Oct 09 '23 16:10 oyhel

It would also be nice to be able to install dependencies via command line which will also speed up installation process.

username-mike avatar Dec 24 '23 19:12 username-mike

Thats really great, I was going to open a request for it. I was thinking of also retaining the installed/required dependencies in the database and when every new worker joins it would auto install all dependencies

saleh-hom avatar Jan 31 '24 09:01 saleh-hom

It would also be nice to be able to install dependencies via command line which will also speed up installation process.

@username-mike and @saleh-hom basically you can do it in two ways.

  1. Press Play button in your spider and type pip install <lib1> <lib2> <lib3> to command input and choose Mode -> All Nodes and after press Confirm btn.

  2. Or you can create requirements.txt with your deps in one of your spiders and run script with the following command for all Nodes: pip install -r requirements.txt

oleksa-dev avatar Mar 05 '24 10:03 oleksa-dev