quicophy
64 thread limit on notebook runs
What is your issue?
I see that there is a hard limit on 64 threads running a notebook of the mdopt project on 96 threaded machine https://github.com/quicophy/mdopt/blob/main/examples/decoding/classical_ldpc.ipynb
not sure if this is (a)typical, so far haven't seen this behaviour before - therefore not exactly sure how to fix this (yet) - haven't seen any obvious hard mutilthreading limits in de code as well - placed this here for future reference as to performance improvements
Hey @twobombs, thanks a lot for reporting this and very sorry for a late response, let me look into it and get back to you shortly!
Hey @twobombs, to be honest I have absolutely zero idea where this limit comes from... Also, where do you see this limit popping up? I see in the screenshot there are 56 processes of which 2 are running.
Since posting this I've looked into he possible cause; one could be that as time progresses and the workload winds down I can see that there is a maximum amount of threads the workload causes by the definition of the workload/algo itself.
I must say running 64+ threads could be regarded as an edge case, but as you mentioned in the UF chat you were looking at speeding up, the dumbest I could think of is adding more threads and I think I bounced into to maximum amount of threads the algo gives work to.
As I scale down with the workload the load (in threads) scales down as well.
The 56 tasks is a top command output and shows the amount of tasks announced in the container the workload is running. At that time only pyton and top are running.