Discussion - Guide to deploy an Elixir/Phoenix app to AWS ECS

[joaquimadraz]

I'm opening this issue for a possible discussion of my blog post: https://joaquimadraz.com/guide-to-deploy-an-elixir-phoenix-app-to-aws-ecs

[tim2CF]

Hello! Thanks for article. I have some questions

1. What's a point to create Distillery releases when you have full-operational Erlang and Elixir inside Docker image (that is used to distribute program)?
2. How distributed Erlang works in case of using of AWS ECS? I mean it seems we need EPMD (Erlang Port Mapper Daemon) configuration and some other magical stuff that will create Erlang nodes cluster from our EC2 instances.
3. How you update your apps in runtime? Any examples of Erlang hot code reloading for EC2 instance?

[joaquimadraz]

Hey @tim2CF, thanks for the feedback.

The Dockerfile is split into two stages, build and release. The build stage uses Distillery to compile the application into Erlang binaries which also includes the Erlang Runtime (I forgot to mention this detail in the article: rel/config.exs#L37). Then release starts with plain Alpine Linux. Nothing from the build phase (meaning Erlang, Elixir, Phoenix, Mix, Node) is carried. There's no Erlang in there until we copy the compiled files into the release image: Dockerfile#L102.

In the end, the release image just has the Erlang runtime and our application binaries.

I still didn't dive into distributed Erlang on AWS. I totally going to use this project to dive into it. Same with hot code reloading. I'll keep you posted.

[tim2CF]

Thanks!

[Merff]

@joaquimadraz Thanks for the guide! I am currently on step docker run --rm -it -p 4002:4002 -e PORT=4002 subs:latest

docker build -t subs:latest \
  --build-arg PHOENIX_SECRET_KEY_BASE=super_secret_phoenix_key_base \ ...

This command works without any errors. But docker run --rm -it -p 4002:4002 -e PORT=4002 subs:latest have error:

Postgrex.Protocol (#PID<0.113.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (localhost:5432): connection refused - :econnrefused
message=><<"connection not available because of disconnection">>

I am new in Docker, like that docker can not connect to the Postgres. May be you know how to fix it?

[joaquimadraz]

Hey @Merff. Yes, what's probably happening is that docker is not able to connect to the host's Postgres. Try using @docker.for.mac.host.internal instead of @localhost on the DATABASE_URL:

DATABASE_URL=postgresql://postgres:[email protected]/subs_prod

Depending on your Docker version, you might need to use @docker.for.mac.host.internal instead of @ localhost for the database url so that the container can access the host’s Postgres. Read more about it here.

There's a note regarding that issue on the guide. I guess it's not properly placed :)

[Merff]

Hi @joaquimadraz. Thanks for the answer) But I don't use Mac, I use Ubuntu. And my project not an Umbrella, just standard Phoenix 1.3 project. How to correct change Dockerfile and may be other files? Thank you!)

[Merff]

Hi! I get "no basic auth credentials" while

docker push "$AWS_ECS_URL"/"$AWS_ECS_DOCKER_IMAGE"

Login works fine with

eval $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)

[markdev]

Hi @joaquimadraz ,

I'm stuck on this step:

$ PORT=4001 _build/prod/rel/subs/bin/subs foreground
>>> Elixir.Subs.Tasks.ReleaseTasks.setup is either not defined or has a non-zero arity

I have module ReleaseTasks in apps/subs/priv/tasks/release_tasks.ex, under defmodule Elixir.Subs.Tasks.ReleaseTasks, with def setup do .... Any idea what could be causing this?

[joaquimadraz]

@Merff sorry for the late reply. Did you figure out what was the problem with the database connection?

Regarding the credentials problem, I'm not sure what can it be but I would say that it's a credentials problem. Make sure that the correct AWS keys are being used. Start by looking at ~/.aws/credentials default credentials.

[joaquimadraz]

@markdev Did you add the priv/tasks to the elixir compilation paths? https://github.com/joaquimadraz/opensubs.io/blob/deployment/apps/subs/mix.exs#L32

[sean-clayton]

UPDATE

I got it all working!!

---

So I followed the steps and everything worked out in the end except... my app doesn't load. I'm completely new to AWS/any kind of devops stuff and am just an application developer.

Here is my project URL: https://gitlab.com/project-bourbon/mash

and here is my public url for the project: http://ec2-54-165-95-3.compute-1.amazonaws.com/

If there's anything glaringly obvious it'd be nice to know. I don't even know how to look at logs or anything :\

Here is my last deploy log as well: https://gitlab.com/project-bourbon/mash/-/jobs/62738365

Locally I can build production and provide the environment variables for my local db and it all works perfectly running from a docker container, so I know it's not that.

[joaquimadraz]

@sean-clayton I'm glad that you were able to do it!

I was checking you project's source code it has something to do with the connection with the database. Do you want to explain a bit the problem you faced?

Regarding the logs, what I do for now is accessing the EC2 instance, grab the docker container logs.

1. ssh to the EC2 instance
2. run docker ps to check the containers that are running

CONTAINER ID	IMAGE	COMMAND	...	PORTS
YourContainerID	subs_web:latest	subs_web/bin/subs...	...	tcp, 0.0.0.0:443->443 ...

3. Grab the container id (YourContainerID) and run docker logs YourContainerID

05:16:51.223 request_id=ar2f8mj2kg6se42lo5sabmgfl0cn6gi5 [info] Sent 200 in 168µs
05:59:35.608 request_id=mqgv4mqcvfgql16jgv9prnaptjce98e8 [info] GET /
05:59:35.608 request_id=mqgv4mqcvfgql16jgv9prnaptjce98e8 [info] Sent 200 in 173µs
06:35:06.920 request_id=5ge7814chchnn91qvvnpuhn75opulo8a [info] HEAD /
06:35:06.920 request_id=5ge7814chchnn91qvvnpuhn75opulo8a [info] Sent 200 in 144µs

This is not ideal but it's a good starting point to check if something went wrong on boot.

I'm working on logging everything to AWS CloudWatch. Would that be interesting for you?

[sean-clayton]

@joaquimadraz Yup, I dug a little bit and forgot that I had ssh access to my EC2 instance 😅 Quickly got access to logs after that. And about CloudWatch: That would definitely interest me. I'm kinda surprised there isn't a step on that in the wizard.

And about the problem I had... I'm not 100% sure why it started working. I think it was because I wasn't passing the DATABASE_URL var as an environment variable so Phoenix couldn't see it. Not 100% sure about that since I wasn't able to look at logs when that was occurring. I then passed down my DB settings in the docker-compose.yml file and it seemed to work after that. Normally, Phoenix would still start even if no DB connection could be made (meaning I could still see my routes and stuff, it would just fail on Repo things), but for me it wasn't even doing that. Then I remembered that I'm running migrations in the before script so I think that those kept failing and preventing Phoenix from even booting up, but these are all just hunches and suspicions.

My next steps that I'm going to try learning (as I said, completely new to this side of development):

• See if WebSockets work and if they don't, try to fix it
• Figure out how to SSL in AWS. Lots of acronyms now... (ACM, EC2, ECS, ELB, etc)
• Stick nginx in front of my ecs cluster to hold onto the certificate

Are these things "good practice" with AWS? Namely wondering about nginx there. I have no idea what the norm is on this field.

---

Update

Got an elastic load balancer with SSL support working :D

[blueangel0723]

@joaquimadraz I have the trouble to deploy the Elixir to AWS. Can you help me, please?

[KalvinHom]

@joaquimadraz Do you have any thoughts on adjusting the deploy.sh to not have the slight downtime between the the old service going down and the new one finishing starting up?

[joaquimadraz]

Hey @KalvinHom. I didn’t have a change to play with it but you can accomplish that using AWS ALB (Application Load Balancer). You can have an ALB pointing to a ECS cluster which allows to do blue/green deployments. After starting the new containers, the ALB will drain the connections from the old containers until all connections are for the new containers. This is as far as I can help you. Haven’t set it up myself.

[stupidusernametoreportissues]

Some minor issues.

What you'll need

• docker
• jq
• awscli
• ecs-cli

1. git clone [email protected]:joaquimadraz/opensubs.io.git
2. cd subs && git checkout 91c25e4 clone creates directory opensubs.io not subs

Now in order for the files under priv/tasks to be compiled and available on the pre_hook_script we need to add the path to elixirc_paths on mix.exs: could add more context such as apps/subs/mix.exs since previously edited root mix.exs

Great job with the commits on deployment branch. Made following along slight ambiguities like this much easier

just noticed oddness https://github.com/joaquimadraz/opensubs.io/blob/deployment/config/deploy/shipit.sh has --region $AWS_ECS_REGION which you corrected elsewhere

[srikanthkyatham]

Hello,

Great article, seems pre_start_hook has been deprecated in favour of pre_start_hooks. Link to changes https://hexdocs.pm/distillery/changelog.html#deprecations

In your blog post set pre_start_hook: "rel/hooks/pre_start.sh" changes to set pre_start_hooks: "rel/hooks"

Hope the future readers benefit from this.

[srikanthkyatham]

Also one other change in the release_tasks.ex

{:ok, _ } = @repo.start_link(pool_size: 1) ->     {:ok, _ } = @repo.start_link(pool_size: 2) // minimum pool size requirement is 2, since ecto 3

Reference http://blog.tap349.com/elixir/ecto/2018/11/15/ecto-upgrading-to-3.0/#increase-connection-pool-size-to-at-least-2

[siyomai]

Hi @joaquimadraz, What would be are changes if we are to use the latest version of distillery? Still figuring things out as far as AWS deployment is concerned. Thanks.

[arjun289]

@siyomai you can consider using the latest version of elixir 1.9.1 which has a releases feature now.

[arjun289]

@sean-clayton were you able to setup nginx with ssl?

[muslimecoin]

how i can do it with multiple images in this project https://github.com/aviacommerce/avia can you give me a explanation on how to deploy it on aws

[chaitanyapi]

Using all the steps followed, i am now able to start my app in EC2 which is having an RDS database. the phoenix endpoint is now my EC2 instance running on 4000 port. I have linked ports 4000 in my docker-compose file and then exposed 4000 port from my container. but still i am not able to connect to the application. I have provided all IP access in the SG. however not sure what am i really missing. any suggestion is highly appreciated. Thank you.

[arzke]

@chaitanyapi I've just run in the same issue. Adding the port 4000 for incoming traffic on my ECS Allowed Ports security group fixed the issue.