gnudatalanguage
shmmap and shmvar does not exist in GDL
should be, er, straightforward for somebody with a modicum of ease in C...
I am coming back to this old issue because I would need it a lot. I am writing programs for multi CPU machines and do not know how I can share the data between the processes without using shmmap and shmvar.
Can anybody work on that or is there any workaround?
Could you please provide an example of usage ?
It is not clear for me, I never had to use that. I also suppose it is totally different from what we do on multi-cores machines ? When I use Slurm or PBS I reserve before the rum some resources, including both CPUs (cores) & RAM ... on a node ..
Are such functions useful nowadays ? (it is a naive question)
Let me show you an easy example. Open two terminals with two IDL sessions.
Session 1:
IDL> shmmap,'anynamehere',/long,5
IDL> tmp=shmvar('anynamehere')
IDL> print,tmp
0 0 0 0 0
Now you open a second session and type the same:
Session 2:
IDL> shmmap,'anynamehere',/long,5
IDL> tmp=shmvar('anynamehere')
IDL> print,tmp
0 0 0 0 0
back to session 1 you fill up the variable:
Session 1:
IDL> tmp(0)=findgen(5)
IDL> print,tmp
0 1 2 3 4
Now on session 2 magically the contents of the variable has changed as well: Session 2:
IDL> print,tmp
0 1 2 3 4
And that is what I need. I want to have access on 80 parallel sessions to the same big array so that all the sessions can work on the same data. Do you have any idea if this can be implemented easily or do you know anything else I can do to share the same array to multiple CPUs?
- thanks for the example, very convenient
- does it have any 'vague) relationship with the "sem" functionalities ? (example in
testsuite/wrong/test_sem.pro)
in fact man shm_open provides examples on my Linux ... And man shmget too ... Which one should we consider ?!
Hi @alaingdl just wanted to know if there is any progress in this issue or any workaround?
FYI, my student @BoudjemaaS just started to have a look on that. Not obvious for us if we are able to do it, but we will try hard, and @Saynkas will help when he will have clarify #1311 :smile:
Thank you for that information! Good luck!
On 31. May 2023, at 10:28, Alain @.***> wrote:
FYI, my student @BoudjemaaS https://github.com/BoudjemaaS just started to have a look on that. Not obvious for us if we are able to do it, but we will try hard, and @Saynkas https://github.com/Saynkas will help when he will have clarify #1311 https://github.com/gnudatalanguage/gdl/issues/1311 😄
— Reply to this email directly, view it on GitHub https://github.com/gnudatalanguage/gdl/issues/860#issuecomment-1570238098, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOC5K6AX2P6HWRGFYNMOMELXI5BRXANCNFSM4U6QOFOA. You are receiving this because you authored the thread.
Hi @alaingdl did you have any progress on this issue? It would be very handy.
Unfortunately no progress. With my student, we did succeed to play with the pure C side of the code without too much problem, but we failed to incorporate it into GDL. It would be great that some experts in C/c++ like @GillesDuvert @slayoo and other give few times on this topic.
Not sure CALL_EXTERNAL could be useful on such topic. Any input welcome.
At first glance this is 'just' having the variable pointer (every GDL variable have such pointer) point to a shared memory region, allocated by mmap() instead of the default allocator. As it goes deep in GDL, and seen the numerous comments in IDL's help page for shmmap, and the need to support all the related functions, it is not 'just' adding a new allocator for variables. But I'm certain it can be managed.
Thanks for having a look.
I would have so many ideas on how to accelerate all my processings using all the CPUs that I have on my machine if that shmmap would work.
Anything that I could help with this issue? Unfortuantely I am not a good C programmer.
On 23. Nov 2023, at 10:59, Giloo @.***> wrote:
At first glance this is 'just' having the variable pointer (every GDL variable have such pointer) point to a shared memory region, allocated by mmap() instead of the default allocator. As it goes deep in GDL, and seen the numerous comments in IDL's help page for shmmap, and the need to support all the related functions, it is not 'just' adding a new allocator for variables. But I'm certain it can be managed.
— Reply to this email directly, view it on GitHub https://github.com/gnudatalanguage/gdl/issues/860#issuecomment-1824485739, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOC5K6HWSOJHY3BMQTUQMU3YF5JFRAVCNFSM4U6QOFOKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGQ2DQNJXGM4Q. You are receiving this because you authored the thread.
@brandy125 & @GillesDuvert based on https://gist.github.com/drmalex07/5b72ecb243ea1f5b4fec37a6073d9d23 I do have a very preliminary working code in GDL, the 2 sessions do exchange memory and I can change the values ...
For sure I will need help to go further. I can send the few files by email (send one to me) ...
As I said in https://github.com/gnudatalanguage/gdl/discussions/1677#discussioncomment-7662589 I'll do that soon. @alaingdl your code changes are welcome.
Hi Alain,
if you have something to test please send me.
Chris
On 24. Nov 2023, at 15:37, Alain @.***> wrote:
based on https://gist.github.com/drmalex07/5b72ecb243ea1f5b4fec37a6073d9d23 I do have a very preliminary working code in GDL, the 2 sessions do exchange memory and I can change the values ...
For sure I will need help to go further. I can send the few files by email ...
— Reply to this email directly, view it on GitHub https://github.com/gnudatalanguage/gdl/issues/860#issuecomment-1825988352, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOC5K6BO4PXQ6XKJOUEW3A3YGDSNNAVCNFSM4U6QOFOKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGU4TQOBTGUZA. You are receiving this because you authored the thread.