Enable multi-stage builds for Autoware

[kaspermeck-arm]

Checklist

• [X] I've read the contribution guidelines.
• [X] I've searched other issues and no duplicate issues were found.
• [X] I've agreed with the maintainers that I can plan this task.

Description

Enable multi-stage builds for Autoware (eg: using Docker Compose or another similar tool).

Purpose

Lay the ground work, with the right tool, to enable building Autoware.Universe as a microservices architecture.

Possible approaches

• Use Docker Compose (this is the tool assumed in the definition of done steps below)

From https://docs.docker.com/compose/: Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Definition of done

• [ ] Create a new script build-compose.sh that uses Docker Compose to build an Autoware container as a multi-stage build (the current local build script is /autowarefoundation/autoware/docker/build.sh)
• [ ] Submit a PR to the Autoware repository with the new build-compose.sh file and request a review from @kenji-miyake
• [ ] Replace the existing build.sh script for local builds
• [ ] Update the existing workflows/actions to use Docker Compose for Docker images built with GitHub Actions
  • [ ] https://github.com/autowarefoundation/autoware/blob/main/.github/workflows/build.yaml
  • [ ] https://github.com/autowarefoundation/autoware/blob/main/.github/workflows/build-self-hosted.yaml
  • [ ] https://github.com/autowarefoundation/autoware/blob/main/.github/actions/docker-build-and-push/action.yaml

[kaspermeck-arm]

@LalithVipulananthan We would need a volunteer who could do the work for this issue, I'll be able to assist of course. From my knowledge, there's only one script which build the actual container image. The files can be found here:

• /home/ubuntu/autoware/docker/build.sh
• /home/ubuntu/autoware/docker/autoware-universe/Dockerfile
• /home/ubuntu/autoware/docker/autoware-universe/docker-bake.hcl

Replacing the build.sh file would be the first step.

[kenji-miyake]

@kasperornmeck cc @LalithVipulananthan FYI, as written in https://github.com/autowarefoundation/autoware/blob/main/docker/README.md, build.sh is for the local build.

The actual images hosted on https://github.com/autowarefoundation/autoware/pkgs/container/autoware-universe are created by these workflows and actions.

• https://github.com/autowarefoundation/autoware/blob/main/.github/workflows/build.yaml
• https://github.com/autowarefoundation/autoware/blob/main/.github/workflows/build-self-hosted.yaml
• https://github.com/autowarefoundation/autoware/blob/main/.github/actions/docker-build-and-push/action.yaml

Replacing the build.sh file would be the first step.

Regarding this part, could you first start with creating it as another script like build-compose.sh? And after checking the behavior and reviewing it, I'd like to consider replacing it.

[kaspermeck-arm]

@kenji-miyake Thanks for sharing these links. I'll have to look more into these to understand the workflow in GitHub.

[kaspermeck-arm]

@kenji-miyake Does this mean that the developer environment container used in the cloud on GitHub is different from the developer container which would be used locally?

[kenji-miyake]

@kasperornmeck No, they are not so different. The differences are just some settings. To build Docker images with GitHub Actions, it's better to use useful third-party actions. But they can't be used in our local, so we need to change some settings to handle the differences.

[ghost]

For the Perception PoC, to build all dependent components for perception, use the following command: colcon build --packages-up-to tier4_perception_launch

[kaspermeck-arm]

@BonoloAWF - I suggest we close this issue as it won't go anywhere and the issues this issue tries to solve is addressed in the DevOps Dojos.