Add tempfs support (feature request)

[mattyweb]

Description

Was trying to deploy our app to ECS and it appeared that tmpfs isn't supported:

WARN[0000] services.tmpfs: unsupported attribute       

My service does data ingestion and has a few tasks that are reading and writing from a tmpfs directory. The final results are committed to a database so I'm happy to have the volume be ephemeral and disappear when the container does.

[chris-crone]

Thanks for the feedback @mattyweb! It looks like tmpfs is not supported by Fargate:

Note: If you are using tasks that use the Fargate launch type, the tmpfs parameter is not supported.

Can you just store your files in /tmp in the container as a workaround? IIRC containers running on Fargate have around 20 GB of free storage?

[mattyweb]

Ah, OK, didn't know that. We use EC2 instances. I gather you're going for lowest common denominator features so it runs the same across EC2, Fargate and Azure?

BTW, not sure if this is the plan but it would be helpful if there were a Docker Compose CLI version of the reference doc here: https://docs.docker.com/compose/compose-file/

In case it's helpful, for starters I just tried running a current docker-compose.yml and got the following:

WARN[0000] services.restart: unsupported attribute      
WARN[0000] services.build: unsupported attribute        
ERRO[0000] service api doesn't define a Docker image to run: incompatible attribute 
ERRO[0000] published port can't be set to a distinct value than container port: incompatible attribute 
WARN[0000] services.restart: unsupported attribute      
WARN[0000] services.build: unsupported attribute        
ERRO[0000] service prefect doesn't define a Docker image to run: incompatible attribute 
WARN[0000] services.tmpfs: unsupported attribute        
WARN[0000] services.restart: unsupported attribute      
WARN[0000] non-external volumes are not supported: unsupported attribute 

I was able to fix everything over time, but took a bit of trial and error.

[chris-crone]

I gather you're going for lowest common denominator features so it runs the same across EC2, Fargate and Azure?

We don't want to make it lowest common denominator as that will likely make the tool too simple for most people. Our approach is to use the existing Compose spec where possible, use infra specific fields when needed (e.g.: x-aws-), and then add to the Compose spec if we see commonality. What we don't want is for the Compose spec to become so specific that users can't deploy a Compose file locally.

We only create an EC2 based ECS cluster if the user requests a GPU; otherwise we use Fargate. This was decided to make the experience as simple as possible for users. You can use an existing cluster with x-aws-cluster.

BTW, not sure if this is the plan but it would be helpful if there were a Docker Compose CLI version of the reference doc here: https://docs.docker.com/compose/compose-file/

This is great feedback, thank you! We decided to keep the docs light while we were iterating quickly but now that we are approaching GA, we need to firm them up. We have a Compose compatibility mapping for ACI that we should do for ECS as well. I've created an issue to track

[nebuk89]

hey @mattyweb, thanks for the feedback. If you would be interested in giving us any more or would be willing to share how you are working with Docker & ECS drop me an email on [email protected]. Would really appreciate more of your input :D Thanks

[ndeloof]

Fargate indeed does not offer tmpfs support (see https://github.com/aws/containers-roadmap/issues/710)

On the other hand Fargate supports ephemeral task-storage volumes, that we could use for comparable purpose. I'm not sure anyway this make sense to consider this an equivalent and silently convert all tmpfs requests into task storages. I'd prefer we make this an opt-in solution, which would be ignored by other implementations (to keep compatibility) using volume long syntax (tmpfs being just a shortcut notation for a volumes entry), would look like this:

services:
  test:
    volumes:
      - type: tmpfs
        size: 1000
        x-aws-task-storage: true