pyspider

pyspider copied to clipboard

I noticed that there is guide of Running pyspider with Docker here.

Actually, it's a little complicated to run so many docker containers at the same time. And there is an alternative advice is using docker-compose, which is not recommended now.

So, I suggest using minikube or kubernetes to manage the set of containers of pyspider and related dependencies, such as mongo, phantomjs, result worker, etc. All we need to do is launching a kubernetes job including all service above.

WDYT @binux