Ability to import docker-compose defintion and convert them as Copilot app and services

[sundarnarasiman]

Docker compose is commonly used for local development and testing. We need an ability for Copilot to import Docker Compose files and convert them as Copilot native app and svc objects, in a guide way. This will be a big boost for developers.

[mikelhamer]

Pretty please

[efekarakus]

Hi @mikelhamer and everyone!

Would it make sense to you if we took each service defined in the Compose file and then create a separate Copilot service with copilot svc init? In this scenario there would be no sidecars for any of your Copilot services.

For images such as postgres would you expect it to translated into an Aurora Serverless PostgreSQL database addon or instead get an ECS service with postgres?

Thanks!

[mikelhamer]

Would it make sense to you if we took each service defined in the Compose file and then create a separate Copilot service with copilot svc init? In this scenario there would be no sidecars for any of your Copilot services.

Yes that would be awesome. I don't even really understand what a sidecar is at this point so I don't have an issue with that (yet) lol.

For images such as postgres would you expect it to translated into an Aurora Serverless PostgreSQL database addon or instead get an ECS service with postgres?

I would expect it to just make an ECS service with postgres. The option to turn it into an RDS cluster could be cool, but I'd rather see that effort go towards expanding the 'storage' functionality :P

This team is awesome!

[srsridharan]

Hi @mikelhamer and everyone!

Would it make sense to you if we took each service defined in the Compose file and then create a separate Copilot service with copilot svc init? In this scenario there would be no sidecars for any of your Copilot services.

Yes that would work well.

For images such as postgres would you expect it to translated into an Aurora Serverless PostgreSQL database addon or instead get an ECS service with postgres?

1. ECS service with postgres to keep it simple. In addition see (2) below for alternative needs that arise.
2. if our dev ops team has setup / maintains the data ware house (e.g. using redshift, or snowflake) would it be possible to specify/map it from the docker file?

Also could we specify the VPC/subnet into which it should be launched?

Ideally for me the goal is to move easily from dev to deployment without needing to take up a lot of dev-ops support.

Thanks for the amazing work on this.

[pharindoko]

@efekarakus any update on this topic ?

[Lou1415926]

Hi @pharindoko! :wave: Thanks for reaching out, we unfortunately don't have a short term plan to progress on this feature request at this moment. Currently, we're working on these features on our sprint board.

Meanwhile, could you tell us more how you'd like this feature to be implemented? For example, would you expect each service in the Compose file to be transformed into an ECS service or would you expect some of them to be a sidecar?

Thanks!

[pharindoko]

Hey @Lou1415926

I would expect them to be sidecars. Otherwise service discovery needs to be enabled.

[piotr-szewczyk]

Is there any news about this issue? Support for docker-compose in aws copilot would really be a big deal, as running docker-compose in AWS leaves a lot to be desired

[huanjani]

Thanks, @Piotrek98, for the nudge!

Can you give us more details on what you're hoping for? As asked above, for a Docker Compose file like

version: "3.9"
   
services:
  db:
    image: postgres
    volumes:
      - ./data/db:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - "8000:8000"
    environment:
      - POSTGRES_NAME=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    depends_on:
      - db

would you expect web (and any other similar listed services) to automatically become a Copilot service? Would you want db to become an ECS Service with postgres or an Aurora Serverless PostgreSQL database add-on under web? Would you want Copilot to guess at the service type (Load Balanced Web Service, Backend, etc.) or would you rather specify it?

This would be a really cool feature; we hope to build it in a way that suits folks' needs!

[cayter]

I think the main ask here is to allow the developers to use copilot + docker-compose.yml to setup preview environments for each branch's pull request. We can always start with the simplest scenario without having a config to map mysql/postgresql to aurora/rds for now.

Would be great if the team can build this into copilot 'cause I'm currently using ecs-cli to setup/teardown the docker-compose.yml ECS cluster for each branch my developers pushing to which can be quite tedious to maintain as it's 2 different tools to maintain.

[david-scottdavomagnifico69]

Any further developments here? And if so how do we use it??

[aprilmintacpineda]

Still can't as of today.

[iamfuzz]

With Docker Compose's integration with ECS retiring next month, this seems even more important now.

[alissaVrk]

in my case, I have 2 services in the docker.compose and I want to deploy both of them on the same machine. basically, the same way that it works locally. without load balancers just some secret and env values

[morganney]

Can the copilot team start with an interactive CLI that parses a docker compose.yml file and then asks how it should be translated into copilot manifests? For instance whether a compose service should be a copilot service, sidecar, addon, etc. and what underlying aws infrastructure to use where applicable (the db for example).

[jj0e]

Many folks are switching to Azure due to deprecation of ecs integration. We need this bad!

[al-dpopowich]

I'll add another voice from the void requesting this feature. We use docker-compose with ecs integration (now deprecated) and are now looking for a new solution without having to jump to cloudformation (gulp) or CDK (please, no) or Terraform or ....

I don't think trying to translate an existing docker-compose service to an AWS service (e.g., postgres to RDS) is what people are asking for, at least not now. We'd just like our docker-compose services to translate to running ECS Fargate services, even if only as sidecars.

The great thing about the now deprecated ecs integration was:

$ docker compose up

Was the command for local development AND deployment. It was a joy, while it lasted, to truly focus on development, not IaC.

[bushwallyta271828]

I haven't tried this repository yet, but after some digging I found https://github.com/compose-x/ecs_composex. Maybe it could help people with broken workflows?