netbird icon indicating copy to clipboard operation
netbird copied to clipboard

Netbird Signal Container does not run on AWS ECS due to missing binaries.

Open joshuahigginson1 opened this issue 1 year ago • 2 comments

Describe the problem When trying to run the dockerhub container on Amazon ECS, the distroless container backing netbird signal does not launch. It errors with an unidentifiable error message (This is a separate AWS issue.)

The container runs absolutely fine after re-building the image with the following Dockerfile:

# Use AWS Bottlerocket as the base image.
FROM amazonlinux:latest

# Set the working directory
WORKDIR /app

# Copy the netbird-signal binary from the original image.
COPY --from=netbirdio/signal:0.25.4 /go/bin/netbird-signal /app/netbird-signal

# Ensure ownership of the file for the nobody user.
RUN chown nobody:nobody /app/netbird-signal && chmod +x /app/netbird-signal

# Set the entry point and default command.
ENTRYPOINT [ "/app/netbird-signal", "run" ]
CMD ["--log-file", "console"]

# Specify a non-root user
USER nobody

Perhaps it would be worth updating the base image to support AWS fargate functionality off the bat. To Reproduce Steps to reproduce the behavior:

  1. Deploy on AWS ECS using the current container held inside of dockerhub - netbirdio/netbird:0.25.4

Expected behavior A clear and concise description of what you expected to happen.

  1. Error logs should stream like so: image

NetBird status -d output: If applicable, add the output of the netbird status -d command

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

joshuahigginson1 avatar Jan 17 '24 12:01 joshuahigginson1

Hi @joshuahigginson1,

I've tested the image (both netbird and signal) on fargate, but both worked fine without issues. The images don't have any dependencies.

Can you share more information about your setup? What was the CPU arch that you used for the task definition?

lixmal avatar Jan 18 '24 11:01 lixmal

Hi @lixmal, we have had no issues whatsoever with the netbird management container. Task def is below. Working totally fine with the dockerfile provided above, but when switching out the signal container for that hosted within dockerhub, no dice.

As no logs were streaming, I believed that it was due to the lack of the awslogs binaries in distroless container.

  ECSTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      # Similar to a name, used for grouping multiple revisions of the task definition.
      Family: "netbird-signal-ecs-task-family"

      Cpu: 512
      Memory: 1024

      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE

      RuntimePlatform:
        OperatingSystemFamily: LINUX
        CpuArchitecture: ARM64

      TaskRoleArn: !GetAtt ECSTaskRole.Arn
      ExecutionRoleArn: !GetAtt ECSTaskExecutionRole.Arn

      ContainerDefinitions:
        - Name: "netbird-signal"
          # Defines that this container is the 'Main' container. If this fails, every other container will stop.
          Essential: true
          Image: !Sub "${NetBirdContainerRepository}:${NetBirdContainerVersion}"
          
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref ECSTaskLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: signal

              # Prevent log loss in event of network error.
              mode: "non-blocking"
              max-buffer-size: "25m"

          # Here, we are telling signal to run GRPC on port 10000. Note that we are using the NLB and AWS' own certificate to perform TLS.
          PortMappings:
            - Name: "signal-grpc-port"
              ContainerPort: 10000
              HostPort: 10000
              Protocol: tcp

          Command: ["--log-file", "console", "--port", "10000"]

joshuahigginson1 avatar Jan 18 '24 12:01 joshuahigginson1

Closing this issue, based on the fix above stable for 2 months.

joshuahigginson1 avatar Mar 14 '24 08:03 joshuahigginson1