PrefectHQ
New worker for running job services in Snowpark Container Services
Describe the current behavior
There is no worker suitable for running flows in Snowpark Container Services.
Describe the proposed behavior
A new worker is added, capable of running flows as job services in Snowpark Container Services.
I've already developed a worker that does this. It isn't ready for release yet, might take a couple of weeks, When it's ready, I'm wondering if it'd make more sense to release it as a PR into prefect-snowflake.
It would be great to hear what someone from the Prefect team thinks. If you're already planning to add this, perhaps my contribution can be a starting point. If not, I'll develop and maintain this on my own, as a 3rd-party package.
Example Use
Users could run their flows in Snowpark Container Services as an alternative to Kubernetes, Azure Container Instances etc.
Additional context
Snowpark Container Services seems to be a Kubernetes-based, Snowflake-managed job service, capable of handling both long-running "services" (with auto restarts etc) as well as short, one-off "job services". Job services seems like the best fit for a Prefect integration.
Hey @bjorhn! It's super cool to hear that you're developing a worker for Snowpark Container Services! I'm definitely open to exploring adding your implementation to prefect-snowflake. When you're ready, feel free to open a PR to prefect-snowflake and we can give it a look!
Also, if you run into any issues while developing the worker, feel free to comment on this issue and we can help!
The worker is done, since yesterday it's running all of our flow runs in our dev and test environments. If it stays stable, we will switch over to it for production work as well, hopefully in about a week. When we are confident it's running well in production, I'll create a PR to gather some feedback.
Hey @bjorhn ! Is your worker implementation in a state that you'd be comfortable sharing? Another group is building out a new worker type soon and I'd love to use your worker as another reference example.
I had to implement a temporary workaround for a problem I've since spoken with Snowflake's engineers about, so my plan was to wait until they push an API update that would let me remove the workaround, which is due in January.
I suppose it wouldn't do any harm to get the PR up anyway, it'll be a good opportunity to collect some feedback meanwhile. We have been running the worker in our production environment for 6 weeks now and it's been near flawless.
I should be able to get something up this weekend.