Feature request: override mount points for each task

[dinvlad]

Hi Team,

Would it be possible to introduce an option to override volume mount path for each task? Right now, we have to register a new task definition just so we could change the mount path on a per-container basis. This introduces additional headache in how to keep track of various definitions, and is also rate limited by the task registration rate.

Our use case is batch processing of various datasets, where the container parameters don't change but the location of the dataset on the local host must be unique for each container/task. E.g. we would have all of our datasets under /datasets/<job_id> on the host, where job_id is unique for each task. One workaround is to pass an environmental variable JOB_ID and have each container read/write to its prefixed location, e.g. if /datasets is mounted to /data in all containers, then each container would gracefully access only /data/<JOB_ID>. However, this provides only minimal isolation of tasks in untrusted environments. I.e. if one task 'misbehaves', it can access everything under /data, including datasets used by other tasks.

It seems like Docker run's -v option maps directly to the containerPath option of a task definition, so is there any reason it cannot be currently overridden for each task, just like environmental variables? We wouldn't even need to change the volume itself, which can remain an immutable part of the task definition.

Thanks

[jahkeup]

@dinvlad thanks for the suggestion and identifying a use case for it! We are tracking this as a feature request.

[tobinski]

Would be nice tho have this feature. I have a similar usecase and need to define separated taskdefinitions for each dataset

[GitShaffi]

@jahkeup We do have a similar usecase. Please let us know there is some progress in this feature request or if there's a workaround for this.

[eoinmurray]

Any update on this @jahkeup - I want to start jupyterlab data-science environments and have each container in its own directory on EFS, so isolating each container like this is really important.

I could register new tasks - but the rate limit of 1/s is too low, any thoughts?

[jsding]

Any progress on this case? we have similar use case. We are creating directories in EFS for each task and mount these directories for each fargate container.

[cutsoy]

Same here. It would make it much easier to run CI jobs in isolation. Each job would mount a different directory from EFS.

[dzerhusen]

Same here would be great for our jupyter environment

[rhlarora84]

Same here would be great for our jupyter environment

+1 for our Jupyter environment too where we use efs for user volumes

[trbabb]

I have a similar case— user data needs to be mapped into an untrusted environment, and the only way to handle this seems to either be to make an unreasonable number of task definitions, or manually run containers on EC2 instances.

What's the likelihood this'll be implemented?

[rhlarora84]

I have a similar case— user data needs to be mapped into an untrusted environment, and the only way to handle this seems to either be to make an unreasonable number of task definitions, or manually run containers on EC2 instances.

What's the likelihood this'll be implemented?

It has only been 4 years of wait ! Should have been a day 1 thing given the docker -v feature.

[trbabb]

I'd do it myself, if I had access to the code! This would save me a ton of trouble.

[dinvlad]

I think your best bet is giving more :+1:s to this issue so it gets a higher priority :wink: Otherwise, it's unlikely to be addressed, just considering the sheer number of open issues here and likely limited resources of the AWS team in charge (I'm not blaming them btw, this is inherently hard; just think how teams work in other companies, it's all very similar).

Or, if you are on paid support, another (maybe even better) way is to advocate with them, so you could get a chance to talk to a PM about your use cases. I've had a number of positive interactions this way (but I can't advocate for this issue anymore, since my current employer is not using AWS much; I would, otherwise).

[sergiy-kozak]

Please prioritize this request, it is enormously useful and naturally maps to docker run -v volume mount options. As already mentioned above, it will resolve tons of pain for us as AWS API users.

[sagard21]

A feature request open for half a decade 😑 . Anyone else here who has found a work around?

[tremendoustj]

Really needed it for one of our use case, but there's no way I could find out to do this. Please prioritize this.