spark-connect-rs icon indicating copy to clipboard operation
spark-connect-rs copied to clipboard

Published 20 hours ago • verified publisher icon sjrusso8

Making Spark Connect Rust more thread-safe friendly

Open edmondop opened this issue 3 months ago • 2 comments

Description

Enabling send_guard and using ARC makes the crate much more friendly to usage in multi-threaded environment, making the session Send and Sync

edmondop avatar Apr 15 '24 13:04 edmondop

Thanks for creating this PR! I played around with something similar when I was rewriting the client implementation. I just didn't like having the user wrap the SparkSession with Arc to create the session. It felt clunky, but that is probably me over thinking the interface.

Do you know of way to allow for friendlier Send and Sync without requiring the user to create Arc? I was loosely mirroring the SessionContext from DataFusion as for how users of a similar DataFrame library interact with a session object. The SessionContext does not get wrapped in Arc.

sjrusso8 avatar Apr 16 '24 14:04 sjrusso8

Will look into it. I think the biggest argument for Arc is to have a cheap copy rather than an expensive one. What do you think? I will look into DataFusion.

On Tue, Apr 16, 2024 at 10:42 AM Steve Russo @.***> wrote:

Thanks for creating this PR! I played around with something similar when I was rewriting the client implementation. I just didn't like having the user wrap the SparkSession with Arc to create the session. It felt clunky, but that is probably me over thinking the interface.

Do you know of way to allow for friendlier Send and Sync without requiring the user to create Arc? I was loosely mirroring the SessionContext from DataFusion https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html as for how users of a similar DataFrame library interact with a session object. The SessionContext does not get wrapped in Arc.

— Reply to this email directly, view it on GitHub https://github.com/sjrusso8/spark-connect-rs/pull/13#issuecomment-2059265371, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGGOKN5OEB7S7F4I6AQLALY5U2GBAVCNFSM6AAAAABGHLMBJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJZGI3DKMZXGE . You are receiving this because you authored the thread.Message ID: @.***>

edmondop avatar Apr 16 '24 19:04 edmondop