pulumi-aws
pulumi-aws copied to clipboard
Published
20 hours ago •
pulumi
pulumi
`aws.batch.JobDefinition` doesn't contain tags when creating a new revision
Describe what happened
We observe this strange behavior with aws.batch.JobDefinition tags:
- When the job definition resource is created it contains tags, be it auto-tags (we use
pulumi-aws-tagslibrary) or tags explicitly defined viatagsargument. - When the job definition resource is modified and the update leads to creating a new revision (e.g. changing the command), the new revision doesn't have the tags. However, the previous (now inactive) revision gets tagged during the update.
- When the job definition resource is modified and the update doesn't lead to creating a new revision (e.g. adding or changing the
tagsargument), the job definition resource receives the tags.
Sample program
import json
import pulumi
import pulumi_aws as aws
from pulumi_aws_tags import register_auto_tags
# Automatically inject tags to created AWS resources.
register_auto_tags(
{"user:Project": pulumi.get_project(), "user:Stack": pulumi.get_stack()}
)
# Use resources managed by infra stack.
infra_stack = pulumi.StackReference("infra-prod")
vpc_id = infra_stack.get_output("vpc_id")
private_subnet_ids = infra_stack.get_output("private_subnet_ids")
# Create an AWS Batch compute environment.
batch_security_group = aws.ec2.SecurityGroup(
"x-tmp-batch-tagging",
vpc_id=vpc_id,
egress=[
aws.ec2.SecurityGroupEgressArgs(
from_port=0, to_port=0, protocol="-1", cidr_blocks=["0.0.0.0/0"]
)
],
)
batch_service_role = aws.iam.Role(
"x-tmp-batch-tagging-service-role",
assume_role_policy=json.dumps(
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {"Service": "batch.amazonaws.com"},
}
],
}
),
managed_policy_arns=[
"arn:aws:iam::aws:policy/service-role/AWSBatchServiceRole"
],
)
compute_environment = aws.batch.ComputeEnvironment(
"x-tmp-batch-tagging",
compute_environment_name_prefix="x-tmp-batch-tagging",
type="MANAGED",
service_role=batch_service_role.arn,
compute_resources=aws.batch.ComputeEnvironmentComputeResourcesArgs(
max_vcpus=1,
security_group_ids=[batch_security_group.id],
subnets=private_subnet_ids,
type="FARGATE",
),
)
aws.ecs.Tag(
"user-project-tag",
resource_arn=compute_environment.ecs_cluster_arn,
key="user:Project",
value=pulumi.get_project(),
)
aws.ecs.Tag(
"user-stack-tag",
resource_arn=compute_environment.ecs_cluster_arn,
key="user:Stack",
value=pulumi.get_stack(),
)
# Create an AWS Batch job queue mapped to the compute environment.
job_queue = aws.batch.JobQueue(
"x-tmp-batch-tagging",
state="ENABLED",
priority=10,
compute_environments=[compute_environment.arn],
)
# Create a log group for the job.
job_log_group = aws.cloudwatch.LogGroup(
"x-tmp-batch-tagging", retention_in_days=30
)
# Create an AWS Batch job definition.
job_execution_role = aws.iam.Role(
"x-tmp-batch-tagging-execution-role",
assume_role_policy=json.dumps(
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {"Service": "ecs-tasks.amazonaws.com"},
}
],
}
),
managed_policy_arns=[
"arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
],
)
job_role = aws.iam.Role(
"x-tmp-batch-tagging-job-role",
assume_role_policy=json.dumps(
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {"Service": "ecs-tasks.amazonaws.com"},
}
],
}
),
)
job_definition = aws.batch.JobDefinition(
"x-tmp-batch-tagging",
type="container",
platform_capabilities=["FARGATE"],
container_properties=pulumi.Output.all(
execution_role=job_execution_role.arn,
job_role=job_role.arn,
log_group=job_log_group.name,
).apply(
lambda args: json.dumps(
{
"command": ["sleep", "15"],
"image": "busybox",
"logConfiguration": {
"logDriver": "awslogs",
"options": {"awslogs-group": args["log_group"]},
},
"runtimePlatform": {
"cpuArchitecture": "ARM64",
"operatingSystemFamily": "LINUX",
},
"resourceRequirements": [
{"type": "VCPU", "value": "0.25"},
{"type": "MEMORY", "value": "512"},
],
"executionRoleArn": args["execution_role"],
"jobRoleArn": args["job_role"],
}
)
),
tags={"user:Foo": "bar"},
propagate_tags=True,
)
# Export stack outputs.
pulumi.export("ecs_cluster", compute_environment.ecs_cluster_arn)
pulumi.export("job_definition", job_definition.name)
pulumi.export("job_queue", job_queue.name)
Log output
Here's the output of pulumi up after changing e.g. command in container properties which leads to a new job definition revision:
Updating (x-tmp-batch-tagging-dev):
Type Name Status Info
pulumi:pulumi:Stack x-tmp-batch-tagging-x-tmp-batch-tagging-dev 2 messages
~ └─ aws:batch:JobDefinition x-tmp-batch-tagging updated (0.59s) [diff: +tags,tagsAll~containerProperties,tags,tagsAll]
Outputs:
ecs_cluster : "xxx"
job_definition: "xxx"
job_queue : "xxx"
Resources:
~ 1 updated
10 unchanged
Duration: 11s
Here's the output after the same change when adding a --diff option:
Updating (x-tmp-batch-tagging-dev):
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:pulumi:Stack::x-tmp-batch-tagging-x-tmp-batch-tagging-dev]
> pulumi:pulumi:StackReference: (read)
[id=infra-prod]
[urn=urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:pulumi:StackReference::infra-prod]
name: "infra-prod"
~ aws:batch/jobDefinition:JobDefinition: (update)
[id=arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:1]
[urn=urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:batch/jobDefinition:JobDefinition::x-tmp-batch-tagging]
[provider=urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:providers:aws::default_6_51_0::b5382c19-dda7-41fd-86a6-43955c2b3f9d]
~ containerProperties: (json) {
~ command : [
[0]: "sleep"
~ [1]: "15" => "10"
]
- environment : []
executionRoleArn : "arn:aws:iam::xxx:role/x-tmp-batch-tagging-execution-role-bd1dc64"
- fargatePlatformConfiguration: {
- platformVersion: "LATEST"
}
image : "busybox"
jobRoleArn : "arn:aws:iam::xxx:role/x-tmp-batch-tagging-job-role-d48642d"
~ logConfiguration : {
logDriver : "awslogs"
options : {
awslogs-group: "x-tmp-batch-tagging-0eb1d94"
}
- secretOptions: []
}
- mountPoints : []
resourceRequirements : [
[0]: {
type : "VCPU"
value: "0.25"
}
[1]: {
type : "MEMORY"
value: "512"
}
]
runtimePlatform : {
cpuArchitecture : "ARM64"
operatingSystemFamily: "LINUX"
}
- secrets : []
- ulimits : []
- volumes : []
}
--outputs:--
~ arn : "arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:1" => "arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:2"
~ containerProperties : (json) {
~ command : [
[0]: "sleep"
~ [1]: "15" => "10"
]
executionRoleArn : "arn:aws:iam::xxx:role/x-tmp-batch-tagging-execution-role-bd1dc64"
fargatePlatformConfiguration: {
platformVersion: "LATEST"
}
image : "busybox"
jobRoleArn : "arn:aws:iam::xxx:role/x-tmp-batch-tagging-job-role-d48642d"
logConfiguration : {
logDriver : "awslogs"
options : {
awslogs-group: "x-tmp-batch-tagging-0eb1d94"
}
secretOptions: []
}
resourceRequirements : [
[0]: {
type : "VCPU"
value: "0.25"
}
[1]: {
type : "MEMORY"
value: "512"
}
]
runtimePlatform : {
cpuArchitecture : "ARM64"
operatingSystemFamily: "LINUX"
}
}
~ id : "arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:1" => "arn:aws:batch:eu-central-1:xxx:job-definition/x-tmp-batch-tagging-23bc289:2"
~ revision : 1 => 2
- tags : {
- user:Foo : "bar"
- user:Project: "x-tmp-batch-tagging"
- user:Stack : "x-tmp-batch-tagging-dev"
}
- tagsAll : {
- user:Foo : "bar"
- user:Project: "x-tmp-batch-tagging"
- user:Stack : "x-tmp-batch-tagging-dev"
}
--outputs:--
ecs_cluster : "xxx"
job_definition: "xxx"
job_queue : "xxx"
Resources:
~ 1 updated
10 unchanged
Duration: 13s
Affected Resource(s)
Output of pulumi about
CLI
Version 3.136.1
Go Version go1.23.2
Go Compiler gc
Plugins
KIND NAME VERSION
resource aws 6.51.0
resource pulumi_aws_tags 1.1.0
language python unknown
Host
OS ubuntu
Version 24.04
Arch x86_64
This project is written in python: executable='/home/xxx/x-tmp-batch-tagging/infra/venv/bin/python' version='3.12.3'
Current Stack: x-tmp-batch-tagging-dev
TYPE URN
pulumi:pulumi:Stack urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:pulumi:Stack::x-tmp-batch-tagging-x-tmp-batch-tagging-dev
pulumi:providers:pulumi urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:providers:pulumi::default
pulumi:pulumi:StackReference urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:pulumi:StackReference::infra-prod
pulumi:providers:aws urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::pulumi:providers:aws::default_6_51_0
aws:iam/role:Role urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:iam/role:Role::x-tmp-batch-tagging-job-role
aws:iam/role:Role urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:iam/role:Role::x-tmp-batch-tagging-service-role
aws:cloudwatch/logGroup:LogGroup urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:cloudwatch/logGroup:LogGroup::x-tmp-batch-tagging
aws:iam/role:Role urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:iam/role:Role::x-tmp-batch-tagging-execution-role
aws:batch/jobDefinition:JobDefinition urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:batch/jobDefinition:JobDefinition::x-tmp-batch-tagging
aws:ec2/securityGroup:SecurityGroup urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:ec2/securityGroup:SecurityGroup::x-tmp-batch-tagging
aws:batch/computeEnvironment:ComputeEnvironment urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:batch/computeEnvironment:ComputeEnvironment::x-tmp-batch-tagging
aws:ecs/tag:Tag urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:ecs/tag:Tag::user-project-tag
aws:ecs/tag:Tag urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:ecs/tag:Tag::user-stack-tag
aws:batch/jobQueue:JobQueue urn:pulumi:x-tmp-batch-tagging-dev::x-tmp-batch-tagging::aws:batch/jobQueue:JobQueue::x-tmp-batch-tagging
Found no pending operations associated with x-tmp-batch-tagging-dev
Backend
Name xxx
URL s3://xxx
User xxx
Organizations
Token type personal
Dependencies:
NAME VERSION
pip 24.2
pulumi_aws_tags 1.1.0
setuptools 75.2.0
wheel 0.44.0
Pulumi locates its logs in /tmp by default
Additional context
We use tags propagation to propagate tags from the job and job definition to the ECS task. We found out in AWS Cost Explorer that the tags of ECS tasks corresponding to Batch jobs started to be missing on May 30 this year. This might relate to the Pulumi AWS package update from 6.34.1 to 6.37.1 that we performed on May 27.
Contributing
Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
Thank you for reporting this strange behavior. The bug reproduces.
Starting with this:
import json
import pulumi as pulumi
import pulumi_aws as aws
test = aws.batch.JobDefinition(
"test",
name="my_test_batch_job_definition",
type="container",
container_properties=json.dumps(
{
"command": [
"ls",
],
"image": "busybox",
"resourceRequirements": [
{
"type": "VCPU",
"value": "1",
},
{
"type": "MEMORY",
"value": "512",
},
]
}
),
tags={"env": "prod"}
)
pulumi.export("name", test.name)
pulumi.export("revision", test.revision)
Checking tagging with this snippet:
aws batch describe-job-definitions --job-definition-name my_test_batch_job_definition |
jq '.jobDefinitions[]|{revision: .revision, tags: .tags}'
After a pulumi up, we get:
{
"revision": 1,
"tags": {
"env": "prod"
}
}
After changing the command and doing another pulumi up:
"command": [
"ls", "--help"
],
We get this:
{
"revision": 1,
"tags": {
"env": "prod"
}
}
{
"revision": 2,
"tags": {}
}
I am finding the same behavior in Terraform proper, so I think this is something that needs fixing ideally in the upstream provider. I will file a bug there and link here.
I have filed this in https://github.com/hashicorp/terraform-provider-aws/issues/39795 - it might help to upvote it to get the maintainers attention! Once the fix is made available the Pulumi provider will likely inherit it and get fixed as well in a regular release.
It looks like the bug fix is released as part of v5.73.0
and pulumi is currently using v.5.75.1, can we please check if the actual pulumi CLI is updated to use this new aws provider version please?
It's already fixed, actually was in a week after TF fix. I confirmed after the Pulumi AWS package update.
I noticed the same with aws:appautoscaling/target:Target
first run, target is created, second run, tags are being applied...
I've confirmed this is fixed in pulumi_aws 6.61.0
@raywonkari can you open a new issue with a repro for aws:appautoscaling/target:Target? Thanks.