kedro-azureml
kedro-azureml copied to clipboard
distributed_job decorator will not accept the shm_size (shared memory parameter)
@distributed_job(Framework.MPI, num_nodes=2, processes_per_node=1)
When using MPI framework and using more nodes in azureml the shared memory (shm_size) is only 2gb by default,
In my usecase, It is giving the memory issue.
I want to increase the shm_size, which was not possible with kedro azureml
But we can pass the shm_size, If we use normal azureml sdk v2 without kedro, while creating the job in the job schema
attaching the example job schema
If I want to have the same in kedro azureml, What is the way?