[BUG] Deploys from a public images not working

[AdrKacz]

Describe the bug I ran ecs-compose-x up -n my-project -f docker-compose.yml, the stack failed with the following errors:

Embedded stack arn:aws:cloudformation:xxx:XXX:stack/xxx-xxx-1234/ABCD was not successfully created: The following resource(s) failed to create: [EcsServiceDefinition].

To Reproduce Steps to reproduce the behavior:

python3 -m venv venv
source venv/bin/activate
pip install pip -U
pip install ecs-composex
pip install "ecs-composex[ecrscan]"

docker-compose.yml

---
version: '3'
services:
  freqtrade:
    image: org/image:stable
    restart: unless-stopped
    container_name: project
    volumes:
      - "./user_data:/freqtrade/user_data"
    ports:
      - "8080:8080"
    # Default command used when running `docker compose up`
    command: >
      start-script
      --logfile /data/user_data/logs/project.log

Expected behavior How can I know what failed in my configuration? It works perfectly fine when running docker compose up so I think I'm using an invalid option.

Thank you for your help

[JohnPreston]

Hello. I am not seeing invalid here in your config. I don't imagine you are using an existing VPC? In the ECS console, if you go and find the service, you will see the tasks being created. There will be an error there that explains what's wrong. Could you try again and report on what's wrong when the tasks start? Thank you :)

EDIT: I use public images all the time. Now, it is possible that Dockerhub will be throttling you, that's something that sometimes does happen.

[AdrKacz]

Thank you for the quick response! Indeed I am not using an existing VPC. I can't see anything in the ECS console, everything got deleted with the AUTO-ROLLBACK (by the time the error fire, the roll-back is already initiated and the resources are deleted) I'm re-deploying and let you know if I see anything.

[AdrKacz]

Great, thank you for your insight, I found out where the error was!

It looks like the files from the volume where not copied, so it fires the following error:

ERROR - Config file "/data/user_data/config.json" not found! Please create a config file or check whether it exists.

This config.json files is located in ./user_data which, when I run locally, is made available thanks to the volume key.

I am not very familiar with Dockers and its deployment. How should I update my docker-compose to use the file from my ./user_data folder?

Afaik, the advantage of using volume is that I can directly update the file while the container run locally, so I have two options to deploy:

1. Either I make the files editable too in the Cloud so that on every deployment I never re-deploy the container, but just the content of the file system
2. I make the file static and I just re-deploy the container with new files

Option 2 looks easier, but I don't know how to proceed. What is your advice?

Thank you for your help ☺️

[JohnPreston]

Indeed, the volumes are extremely useful. I use them myself all the time. The trouble with this is, it works great locally, and it works great on ECS, but how do you get the file from your local disk, to AWS.

Plus, currently I have often faced that you can't mount just the file itself, you need to mount, say config:/_config and find a way to put your config file in the /_config/ path.

Something that I also created, because I have so many services that need dynamically generated content or simply, a config file or whatever it is that you want to have not in the docker image, is ECS Files Composer (and 37M downloads? I didn't even realize :face_with_open_eyes_and_hand_over_mouth: )

So, usually when I have a local file like you do that you need mounting, I make sure to put it in a local directory (folder) and mount the directory, not the file. You never know, you might need config files later.

I use SSM parameters (but, there is a size limit so S3 might be a better choice) to upload the content of the file. Then, files-composer will pick that up first, and store it in a docker volume shared between the two containers.

If files-composer was successful, then will the application container starts. This is quite useful. In this file you will see how the two services are set up, how the ssm parameter comes into play: prometheus needs the config file. the files composer will pull it from SSM. Render the file into a shared volume. Then prometheus starts, and finds the file.

Hope that helps, let me know how you get on.