pydelhi
Building backends using `entry_points` and NetworkX's parallel backend
Title
nx-parallel : NetworkX's parallel backend
Abstract:
Hi! Join me as we explore parallel graph algorithms in nx-parallel, going over the implementation details, performance enhancements, the effects of chunking, and challenges of parallelizing functions that “yield” instead of “return”. We’ll start by exploring a social network analysis problem and see the need for a parallel backend. Then we’ll dive into the NetworkX’s backend dispatching architecture and walk through setting up a custom backend and testing it using the existing suite, and understand the significance of the dispachable decorator. We'll delve into the internals of parallel algorithms, observing process creation and CPU core utilization. Finally, we'll discuss future directions for handling even larger graphs. Experience how nx-parallel accelerates network analysis through parallel processing!
Description:
During the first 5-6 minutes of the talk, I will guide the audience through a comprehensive, but brief, journey into the world of network analysis through a social networks example. I’ll begin by performing the computations using NetworkX and then move on to the need for a parallel backend. The initial segment will unravel some of the core NetworkX algorithms and backend dispatching, shedding light on the automated testing framework and the significance of the dispatchable decorator.
Transitioning into the crux of the discussion, the subsequent 10-15 minutes will be dedicated to dissecting how parallel algorithms are implemented in nx-parallel. I will elucidate the internal workings of parallelization, showcasing the increase in processes and the activity distribution across different CPU cores while the parallel algorithms are running. A focal point of this section will be the role of chunking in nx-parallel and its impact on speedups. Furthermore, I may present benchmarking results that demonstrate the performance benefits derived from parallel computing. Moreover, I will also address challenges encountered while parallelizing functions that yield instead of return a value and what chunking means for this kind of function. By sharing my experiences of fiddling with these issues, attendees might gain some insights into navigating similar hurdles in their own projects.
In the closing minutes, we'll take a glimpse into the future of nx-parallel and NetworkX dispatching, by discussing some of the “bigger picture” questions about the pipeline connecting NetworkX to nx-parallel to joblib to joblib's different parallel backends and using something like distributed graph processing for a much more memory-efficient parallelization. And then this section will lead to a much more interactive 5-minute window for the audience to participate, ponder and raise more questions, and share their thoughts, knowledge, and perspectives!
Pre-requisites & reading material
None, Not required but if you want to read: https://schefflera-arboricola.github.io/Schefflera-Arboricola/NetworkX-Internship-Working-on-nx-parallel
Time required for the talk
~ 30 mins
Link to slides/demos
No response
About you
Hi, I'm Aditi. I am currently an undergraduate student. I'm doing my bachelor's in Computer Science from Delhi University, and my bachelor's in Data Science and Application from the Indian Institute of Technology, Madaras. I've been working on the NetworkX project and the nx-parallel backend for quite a few months now, previously as an Independent contractor and currently as a GSoC contributor.
Blog: https://schefflera-arboricola.github.io/Schefflera-Arboricola/ GitHub: https://github.com/Schefflera-Arboricola
Availability
June'24-Aug'24
Illustration
nx-parallel github repo: https://github.com/networkx/nx-parallel
Hi @Schefflera-Arboricola thanks for the submission! Will you be available to present this talk on 16th March? This meetup is a collaborative one with Neo4j (More details here)
Hi, I've updated my availability above. Please let me know of any relevant events in those days. And, sorry for the delayed response.
Thank you :)
Hi @Schefflera-Arboricola, we're having a meetup on 15th June (3rd Saturday). Would you be available to present?
Would you be available to present?
Yes
Please let me know the event's details and the audience's background and knowledge. Feel free to contact me via email: [email protected]
@Schefflera-Arboricola we're postponing the meetup to 22nd June. This is a combined meetup (almost a mini-conference) with @rustdelhi and @ILUGD. We already have 2 speakers from PyDelhi.
I'd suggest you attend the meetup and give this as a lightning talk (5-10 mins) if possible. You can also give it in the July meetup (full 30 mins), no issues.
I'd suggest you attend the meetup and give this as a lightning talk (5-10 mins) if possible. You can also give it in the July meetup (full 30 mins), no issues.
I can give this as a lightning talk on 22nd. Please LMK the next steps. And I would be open to giving a 30 min talk as well in July but please LMK the dates before confirming. Thanks!
Scheduling this talk as a lightning talk for 22nd June. There are no next steps as such. You just have to show up and deliver the talk :). The timings are 10am-1pm and there'll be a dedicated time slot for lightning talks. So, do keep checking the meetup.com page or our soical media page for announcements.
I'll also post the timing here once everything is finalized.
@pulsar17 Will I be able to screen share from my own laptop and present my talk? Thanks!
Yes!
Best Ishaan Arora
A man’s not dead while his name is still spoken.
-------- Original Message -------- On 21/06/24 9:33 am, Aditi Juneja wrote:
@.***(https://github.com/pulsar17) Will I be able to screen share from my own laptop and present my talk? Thanks!
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
Thanks for giving the talk! Do you want to present again in July (full 30 mins)?
Thank you for the opportunity! And yes I would like to present it as a 30 min talk in July :)
@Schefflera-Arboricola would you be able to present this on 20 July?
yes! please LMK the timings and venue. Thanks :)
The venue is Sun foundation World class skill center (Blue Line). We haven't announced it yet. The timings will be 1-5 pm.
See you there!
