Charles Stern
Charles Stern
What versions of Beam + Python are you running? Possibly related to https://github.com/pangeo-forge/pangeo-forge-runner/issues/78?
I routinely use multithreading on the direct runner (never used interactive) without issue. Not to mention that we also do this [in our integration tests, which are passing](https://github.com/pangeo-forge/pangeo-forge-recipes/actions/runs/6264387376/job/17010968525)! 😄 I...
> (b) direct_running_mode doesn't actually do anything. In experimenting with the example Tom provides myself, my conclusion is that `direct_running_mode` definitely does appear to change the running behavior of local...
> I agree that it does not appear based on that example that multi-processing takes any less wall time than single threaded execution @TomNicholas and I suspected that if the...
I agree we'll still want to wait for `beam-refactor` to go in before approaching this, and the following is not necessarily a drop-in fix for this, but noting what seems...
Thanks for the clarifications, @martindurant!
Related work: - CLI passthrough: https://github.com/pangeo-forge/pangeo-forge-runner/pull/106 - Action branch using this CLI passthrough: https://github.com/pangeo-forge/deploy-recipe-action/tree/storage-cls-args - Real world deployment test: https://github.com/pangeo-forge/aqua-modis-feedstock/pull/7
@chuckwondo thanks for using Pangeo Forge and reporting back on your experience. There are definitely still some unpolished edges and these types of reports are invaluable for surfacing areas we...
@chuckwondo, way to push this to a working state! Responses inline: > Now my question is, what do I need to do from here to make a good submission for...
Thanks all for your work on this so far! Reviving this thread as today @TheNeuralBit discovered that it is the root cause of https://github.com/apache/beam/issues/29365 (thanks Brian!). To summarize the context...