Bhimraj Yadav
Bhimraj Yadav
Hi @lantiga, The PR is ready for review. Thank you!
> @bhimrazy great idea. want to take a stab at submitting a PR? we can help you finish and land it. Sure @williamFalcon. Sounds great.
Thank you, @AugustDev, for the insights and solution. Closing this issue, as the behaviour seems to be expected in a multi-worker setting. cc: @tchaton
Let’s keep this open for now — could be fun to try out for learning purposes. Also, cool to see zstd coming to the Python 3.14 stdlib! 🚀 https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-pep784
Let's keep this. interesting one.
I think this issue will be interesting for new contributors once we have a clear breakdown of it. I’ll also brainstorm a bit more on it. One simple idea that...
Ah, I see now—this actually happens at stream time, not during optimization as I initially thought. Thanks for clarifying that, @deependujha.
Could this still be an issue? It also seems like the large number of small datasets might be the bottleneck here for the threads.
Let's keep this open, as we've been experimenting around this issue. We'll continue exploring and will add our findings here from the last few experiments.
**Idea**: One of the other ideas that could be explored is this: we create a multiprocessing dictionary and share it between the workers or a simple dict to keep within...