Inconsistent usage of awslogs_stream_prefix

[pyrr]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.29.0

Apache Airflow version

2.9.2

Operating System

Amazon Linux 2023.5.20240805

Deployment

Amazon (AWS) MWAA

Deployment details

Using a Python 3.11 environment with botocore==1.34.106 and apache-airflow-providers-amazon==8.29.0.

What happened

The code for EcsRunTaskOperator looks for log streams by using the pattern prefix-name/ecs-task-id:

    def _get_logs_stream_name(self) -> str:
        return f"{self.awslogs_stream_prefix}/{self._get_ecs_task_id(self.arn)}"

In the AWS ECS docs for LogConfiguration, log streams are created using the pattern prefix-name/container-name/ecs-task-id.

What you think should happen instead

EcsRunTaskOperator should follow the same log stream naming convention as specified in the AWS ECS docs.

Because this isn't the case, you need to specify a stream prefix using a different pattern when registering your task vs. when running your task.

How to reproduce

1. Register a task definition with a containerDefinition containing at least the following:

  "name": "my-container-name",
  "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
          "awslogs-group": "my-log-group",
          "awslogs-region": "us-east-1",
          "awslogs-create-group": "true",
          "awslogs-stream-prefix": "my-container-prefix",
      },
  },

The logs in your ECS task will be uploaded to CloudWatch under the log group my-log-group and the prefix my-container-prefix/my-container-name/task-id.

2. Run the task using the EcsRunTaskOperator with the same awslogs-stream-prefix.
3. View the logs in Airflow-- they will fail to fetch the task logs from CloudWatch because they are looking for the log group my-log-group and prefix my-container-prefix/task-id.

Anything else

The only workaround for this issue is to specify the task definition's awslogs-stream-prefix with my-prefix-name and the EcsRunTaskOperator's awslogs_stream_prefix with my-prefix-name/my-container-name.

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct