redun
redun copied to clipboard
insitro
Google cloud
Hi! this is quite a clever pythonic and readable workflow manager! thank you for it.
If I understand correctly, your cloud support is limited to AWS (AWS Batch, AWS Glue). Any plans to add GCP support?
Hi @jshleap, thanks for question.
We currently support all the object storage systems provided by fsspec, which include GCS. As for compute, the executor design should allow many different backends over time. The closest to executor to being useful in GCP might be the kubernetes executor being worked on in #22. Let me know if that helps.
Not quite, unless image can be a GCP uri? The ideal scenario would be that given an image and an storage uris, you could run "batch" in either of the cloud providers. But I understand that it could mean a lot of work.
Thanks for the answer.
Ah, great point. The Docker repository would need to be generalized as well.
So far we've been prioritizing work as it comes up naturally in our own projects, but PRs are welcomed as well. Thanks again for the ideas.
Just to give an update here. Our docker uri can be any docker registry, so there isn't anything ECR specific.
We also have generalized our File staging to support GCS and other fsspec supported filesystem.
We are still working on #22 for providing an actual executor.
@mattrasmus I'm curious if anyone is currently working on implementing a Google Batch executor for redun.
I am adding this here as a reference https://github.com/insitro/redun/pull/66 Seems like GCP batch has already been added
