kedro-azureml icon indicating copy to clipboard operation
kedro-azureml copied to clipboard

distributed_job decorator will not accept the shm_size (shared memory parameter)

Open snavyareddy opened this issue 2 months ago • 0 comments

@distributed_job(Framework.MPI, num_nodes=2, processes_per_node=1)

When using MPI framework and using more nodes in azureml the shared memory (shm_size) is only 2gb by default, In my usecase, It is giving the memory issue. image

I want to increase the shm_size, which was not possible with kedro azureml

But we can pass the shm_size, If we use normal azureml sdk v2 without kedro, while creating the job in the job schema attaching the example job schema image

If I want to have the same in kedro azureml, What is the way?

snavyareddy avatar May 08 '24 15:05 snavyareddy