Charles Stern

What versions of Beam + Python are you running? Possibly related to https://github.com/pangeo-forge/pangeo-forge-runner/issues/78?

I routinely use multithreading on the direct runner (never used interactive) without issue. Not to mention that we also do this [in our integration tests, which are passing](https://github.com/pangeo-forge/pangeo-forge-recipes/actions/runs/6264387376/job/17010968525)! 😄 I...

> (b) direct_running_mode doesn't actually do anything. In experimenting with the example Tom provides myself, my conclusion is that `direct_running_mode` definitely does appear to change the running behavior of local...

> I agree that it does not appear based on that example that multi-processing takes any less wall time than single threaded execution @TomNicholas and I suspected that if the...

I agree we'll still want to wait for `beam-refactor` to go in before approaching this, and the following is not necessarily a drop-in fix for this, but noting what seems...

Thanks for the clarifications, @martindurant!

Related work: - CLI passthrough: https://github.com/pangeo-forge/pangeo-forge-runner/pull/106 - Action branch using this CLI passthrough: https://github.com/pangeo-forge/deploy-recipe-action/tree/storage-cls-args - Real world deployment test: https://github.com/pangeo-forge/aqua-modis-feedstock/pull/7

@chuckwondo thanks for using Pangeo Forge and reporting back on your experience. There are definitely still some unpolished edges and these types of reports are invaluable for surfacing areas we...

@chuckwondo, way to push this to a working state! Responses inline: > Now my question is, what do I need to do from here to make a good submission for...

Thanks all for your work on this so far! Reviving this thread as today @TheNeuralBit discovered that it is the root cause of https://github.com/apache/beam/issues/29365 (thanks Brian!). To summarize the context...