Automatically cluster with AWS execution environments

[krainboltgreene]

This will pick out the parts needed to cluster for any livebook application deployed on the AWS execution platforms.

• [ ] Write documentation describing cloud map setup
• [ ] Write documentation describing port setup
• [ ] Write documentation describing security groups
• [ ] Write documentation for partial automation

[CLAassistant]

All committers have signed the CLA.

[krainboltgreene]

Good news: There's like 5-6 different jq alternatives. Bad news: They're all focused on speed/language availability rather than size.

One option we have is to utilize a call to the Elixir shipped binary to parse/query?

[josevalim]

Oh, calling the Elixir binary is a good call. Let's do that. You can also call erl and use the new built-in json parser.

[jonatanklosko]

@josevalim calling elixir and :json sounds best (otherwise we need to mix.install), but currently we only require OTP 25. That said, it should be fine to bump the requirement, since for Desktop and Docker we always use latest anyway.

[krainboltgreene]

@jonatanklosko One of Livebook's deps is jason, so I used that instead to avoid bumping requirements.

[krainboltgreene]

172.31.49.31root@ip-172-31-metadata=`curl --silent ${ECS_CONTAINER_METADATA_URI}`TADATA_URI}`
root@ip-172-31-49-31:/app# machine_ip=`echo $metadata | bin/core eval 'IO.read(:stdio, :eof) |> Jason.decode!() |> Map.get("Networks") |> Enum.at(0) |> Map.get("IPv4Addresses") |> Enum.at(0) |> IO.write()'`
root@ip-172-31-49-31:/app# image_id=`echo $metadata | bin/core eval 'IO.read(:stdio, :eof) |> Jason.decode!() |> Map.get("ImageID") |> IO.write()'`
root@ip-172-31-49-31:/app# deployment="$(echo $image_id | shasum | cut -c 1-10)"
root@ip-172-31-49-31:/app# echo "livebook-${deployment}@${machine_ip}"
[email protected]

Here it is working.

[krainboltgreene]

Okay so the remaining work here is the documentation of Cloud Map and Security Groups. That said this is actually rather generic to Phoenix itself, or even Elixir, so should I suggest it as documentation to those projects?

[josevalim]

I'd document it on Livebook first, for two reasons:

1. Elixir itself does not have a specific page for documenting per provider option. There are just too many possible scenarios for the language docs to cover.

2. Phoenix has a section on deployment, which you could add this, but you would need to cover all of the aspects of deployment, which will be more work.

WDYT?

[krainboltgreene]

Roger, will start today.

For the future I think maybe there's a signal here telling us that an elixir cookbook, community run like Bevy's but outside the scope any framework or the language, might be worth organizing.

[jonatanklosko]

@krainboltgreene good call parsing with release eval!

[github-actions[bot]]

Uffizzi Preview deployment-63204 was deleted.

[jonatanklosko]

Looks good to me, we just need the docs now. Ideally with some IaC snippets, I think either is fine, terraform, cloud formation or CLI, just so it's more standardised.

[krainboltgreene]

Back from the dead I finally had the time to finish off this meager documentation describing as much as possible without getting too much into the weeds.

[josevalim]

:green_heart: :blue_heart: :purple_heart: :yellow_heart: :heart: