PutBucketLifecycleConfiguration writing fails with MalformedXML

[rabarar]

Describe the bug

when attempting to Put a lifecycle configuration onto a bucket, it fails with:

Got an error creating the policy:
operation error S3: PutBucketLifecycleConfiguration, https response error StatusCode: 400, RequestID: M2506HYABH1AT7BK, HostID: ttF1JGprIhRDwOkz8e22C4EQR3CoUFvWyFau4z6fckhzzOH9aIGB25HtqZH7cxPy1DOTqD5lzlc=, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

The input looks as follows:

  input := &s3.PutBucketLifecycleConfigurationInput{
        Bucket: bucket,
        LifecycleConfiguration: &types.BucketLifecycleConfiguration{
            Rules: []types.LifecycleRule{
                {       
                    Status: types.ExpirationStatusEnabled,
                    AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
                        DaysAfterInitiation: 1,
                    },  
                    Expiration: &types.LifecycleExpiration{
                        ExpiredObjectDeleteMarker: true,
                        Days:                      0,
                        Date:                      nil,
                    },
                    NoncurrentVersionExpiration:  nil,
                    NoncurrentVersionTransitions: []types.NoncurrentVersionTransition{},
                    ID:                           &lifecyleID,
                    Prefix:                       nil,
                    Transitions:                  []types.Transition{},
                },
            },
        }, 
    }

Expected Behavior

Expect it to write the lifecycle configuration with the instantiated input values

Current Behavior

failure with:

Got an error creating the policy:
operation error S3: PutBucketLifecycleConfiguration, https response error StatusCode: 400, RequestID: M2506HYABH1AT7BK, HostID: ttF1JGprIhRDwOkz8e22C4EQR3CoUFvWyFau4z6fckhzzOH9aIGB25HtqZH7cxPy1DOTqD5lzlc=, api error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema

Reproduction Steps

Use this input:

   input := &s3.PutBucketLifecycleConfigurationInput{
        Bucket: bucket,
        LifecycleConfiguration: &types.BucketLifecycleConfiguration{
            Rules: []types.LifecycleRule{
                {   
                    Status: types.ExpirationStatusEnabled,
                    AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
                        DaysAfterInitiation: 1,
                    },  
                    Expiration: &types.LifecycleExpiration{
                        ExpiredObjectDeleteMarker: true,
                        Days:                      0,  
                        Date:                      nil,
                    },  
                    NoncurrentVersionExpiration:  nil,
                    NoncurrentVersionTransitions: []types.NoncurrentVersionTransition{},
                    ID:                           &lifecyleID,
                    Prefix:                       nil,
                    Transitions:                  []types.Transition{},
                },  
            },  
        },  
    }   

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

"github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/s3" "github.com/aws/aws-sdk-go-v2/service/s3/types"

Compiler and Version used

go version go1.18.3 darwin/arm64

Operating System and version

Mac OSX 12.4

[skmcgrail]

Could you follow the instructions in the Developer Guide to enable client logging with the aws.LogRequestWithBody option enabled? That would help us understand the serialized XML that is being sent to the service for your particular request. Thanks!

[rabarar]

Here’s the serialized XML on the request body:

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><AbortIncompleteMultipartUpload><DaysAfterInitiation>1</DaysAfterInitiation></AbortIncompleteMultipartUpload><Expiration><ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker></Expiration><ID>cleanup853211</ID><Status>Enabled</Status></Rule></LifecycleConfiguration>

On Jun 8, 2022, at 12:44 PM, Sean McGrail @.***> wrote:

Could you follow the instructions in the Developer Guide https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/logging/ to enable client logging with the aws.LogRequestWithBody option enabled? That would help us understand the serialized XML that is being sent to the service for your particular request. Thanks!

— Reply to this email directly, view it on GitHub https://github.com/aws/aws-sdk-go-v2/issues/1722#issuecomment-1150151243, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABB2J6HWUPSKYUINIBMK2VTVODEWNANCNFSM5X7GOUEQ. You are receiving this because you authored the thread.

[skmcgrail]

Thank you for the logging information @rabarar ,

I was also able to reproduce this on my end as well in both the V2 and V1 Go SDKs, and the AWS CLI. Will follow-up this investigation with the service team.

[skmcgrail]

I've root caused this down to the service expecting the filter API member to be serialized to the request regardless if sent or not. If you provide an empty filter at the CLI it will work correctly, same with the V1 SDK (provide a pointer to a structure).

Works

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Rule>
        <AbortIncompleteMultipartUpload>
            <DaysAfterInitiation>1</DaysAfterInitiation>
        </AbortIncompleteMultipartUpload>
        <Expiration>
            <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
        </Expiration>
        <Filter />
        <ID>abortMulti</ID>
        <Status>Enabled</Status>
    </Rule>
</LifecycleConfiguration>

Doesn't Work

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Rule>
        <AbortIncompleteMultipartUpload>
            <DaysAfterInitiation>1</DaysAfterInitiation>
        </AbortIncompleteMultipartUpload>
        <Expiration>
            <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
        </Expiration>
        <ID>abortMulti</ID>
        <Status>Enabled</Status>
    </Rule>
</LifecycleConfiguration>

Unfortunately the Go V2 SDK uses a union type, so you can't quite set this in exactly the same way.

[rabarar]

as an aside, attempting to reference the Filter type LifecycleRuleAndOperator in types fails to resolve and is inconsistent with the go-docs.

On Jun 9, 2022, at 12:52 PM, Sean McGrail @.***> wrote:

I've root caused this down to the service expecting the filter API member to be serialized to the request regardless if sent or not.

Works

1 true abortMulti Enabled Doesn't Work 1 true abortMulti Enabled — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you were mentioned.

[skmcgrail]

This code snippet does appear to work, though the serialization is different. It would provide a temporary work around at least:

_, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3.PutBucketLifecycleConfigurationInput{
		Bucket: aws.String("mcgrails-test-data"),
		LifecycleConfiguration: &types.BucketLifecycleConfiguration{
			Rules: []types.LifecycleRule{
				{
					Status: types.ExpirationStatusEnabled,
					AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: 1,
					},
					Expiration: &types.LifecycleExpiration{
						ExpiredObjectDeleteMarker: true,
					},
					Filter: &types.LifecycleRuleFilterMemberPrefix{Value: ""},
					ID:     aws.String("abortMulti"),
				},
			},
		},
	})

This serializes as:

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
    <AbortIncompleteMultipartUpload>
        <DaysAfterInitiation>1</DaysAfterInitiation>
    </AbortIncompleteMultipartUpload>
    <Expiration>
        <ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
    </Expiration>
    <Filter>
        <Prefix></Prefix>
    </Filter>
    <ID>abortMulti</ID>
    <Status>Enabled</Status>
</Rule>
</LifecycleConfiguration>

[skmcgrail]

This seems to be broken in the S3 Control model as well, for different, currently unknown, reasons...

client := s3control.NewFromConfig(cfg)
_, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3control.PutBucketLifecycleConfigurationInput{
	AccountId: aws.String("accountId"),
	Bucket:    aws.String("mcgrails-test-data"),
	LifecycleConfiguration: &types.LifecycleConfiguration{
		Rules: []types.LifecycleRule{
			{
				Status: types.ExpirationStatusEnabled,
				Filter: &types.LifecycleRuleFilter{},
				AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
					DaysAfterInitiation: 1,
				},
				Expiration: &types.LifecycleExpiration{
					ExpiredObjectDeleteMarker: true,
				},
				ID: aws.String("abortMulti"),
			},
		},
	},
})

Returns

<?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse>
    <Error>
        <Code>InvalidURI</Code>
        <Message>Couldn't parse the specified URI.</Message>
        <URI>bucket/mcgrails-test-data/lifecycleconfiguration</URI>
    </Error>
    <RequestId>Z2AGJDNK1QRRVC4V</RequestId>
    <HostId>3pXY5FhtMiP+Bq0w6wEU786mNkWl4ROwekWZ9JOusTZ5UlKkZZictU0+FkpbZlVwY2oa43KmzdQ=</HostId>
</ErrorResponse>

Seeing the same behavior with the Go V1 SDK and CLI.

[rabarar]

Is there a timeline for a fix? Feel free to send me a t-shirt for finding this one ;)

On Jun 9, 2022, at 4:01 PM, Sean McGrail @.***> wrote:

 This seems to be broken in the S3 Control model as well, for different, currently unknown, reasons...

client := s3control.New(sess) _, err := client.PutBucketLifecycleConfiguration(&s3control.PutBucketLifecycleConfigurationInput{ AccountId: aws.String(""), Bucket: aws.String("mcgrails-test-data"), LifecycleConfiguration: &s3control.LifecycleConfiguration{ Rules: []*s3control.LifecycleRule{ { AbortIncompleteMultipartUpload: &s3control.AbortIncompleteMultipartUpload{ DaysAfterInitiation: aws.Int64(1), }, Expiration: &s3control.LifecycleExpiration{ ExpiredObjectDeleteMarker: aws.Bool(true), }, Filter: &s3control.LifecycleRuleFilter{}, ID: aws.String("abortMulti"), Status: aws.String("Enabled"), }, }, }, }) Returns

InvalidURI Couldn't parse the specified URI. bucket/mcgrails-test-data/lifecycleconfiguration Z2AGJDNK1QRRVC4V 3pXY5FhtMiP+Bq0w6wEU786mNkWl4ROwekWZ9JOusTZ5UlKkZZictU0+FkpbZlVwY2oa43KmzdQ= Seeing the same behavior with the Go V1 SDK and CLI.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[rabarar]

Definitely works but has zero relationship to the docs … not sure where types.LifecycleRuleFilterMemberPrefix comes from …

On Jun 9, 2022, at 2:00 PM, Sean McGrail @.***> wrote:

This code snippet does appear to work, though the serialization is different. It would provide a temporary work around at least:

_, err = client.PutBucketLifecycleConfiguration(context.TODO(), &s3.PutBucketLifecycleConfigurationInput{ Bucket: aws.String("mcgrails-test-data"), LifecycleConfiguration: &types.BucketLifecycleConfiguration{ Rules: []types.LifecycleRule{ { Status: types.ExpirationStatusEnabled, AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{ DaysAfterInitiation: 1, }, Expiration: &types.LifecycleExpiration{ ExpiredObjectDeleteMarker: true, }, Filter: &types.LifecycleRuleFilterMemberPrefix{Value: ""}, ID: aws.String("abortMulti"), }, }, }, }) This serializes as:

1 true abortMulti Enabled — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you were mentioned.

[RanVaknin]

Hi @rabarar ,

You have to specify the <Filter> tag unless you have a <Prefix> tag in your xml.

source:

Filter The Filter is used to identify objects that a Lifecycle Rule applies to. A Filter must have exactly one of Prefix, Tag, or And specified. Filter is required if the LifecycleRule does not contain a Prefix element. Type: LifecycleRuleFilter data type

I think this isn't a bug, just an API design choice from s3.

[rabarar]

Hi -The API doesn’t afford access to the xml - that is internal to the go api and therefore is a bug imho.Best,RobOn Nov 30, 2022, at 6:03 PM, Ran Vaknin @.***> wrote: Hi @rabarar , You have to specify the <Filter> tag unless you have a <Prefix> tag in your xml. source:

Filter The Filter is used to identify objects that a Lifecycle Rule applies to. A Filter must have exactly one of Prefix, Tag, or And specified. Filter is required if the LifecycleRule does not contain a Prefix element. Type: LifecycleRuleFilter data type

I think this isn't a bug, just an API design choice from s3.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[RanVaknin]

@rabarar ,

After taking a deeper look into this here is what I have found.

The SDK serializes Golang strcuts into XML. When you send a request without that field in the request you're essentially leaving it out of the XML. When the service reads the XML body sent by the GO SDK Client and sees that your LifeCycleRule doesn't have a <Filter> tag, it looks for <Prefix> type and if both are missing it throws this exception.

This is all handled on the service-side, and is documented.

To show that its not a GO SDK bug I've tested it on JS SDK V3 with the following code:

❌ Incorrect:

import { S3Client, PutBucketLifecycleConfigurationCommand } from '@aws-sdk/client-s3';

const client = new S3Client({ region: 'us-east-1' });

const command = new PutBucketLifecycleConfigurationCommand({
    Bucket: 'mybucket123',
    LifecycleConfiguration: {
        Rules: [
            {
                Status: 'Enabled',
                ID: 'abortMulti',
                Expiration: {ExpiredObjectDeleteMarker: true},
            }
        ]
    }
});

try {
    const data = await client.send(command);
    console.log(data)
} catch (error) {
    console.log(error);
}

❌ Incorrect:

const command = new PutBucketLifecycleConfigurationCommand({
    Bucket: 'mybucket123',
    LifecycleConfiguration: {
        Rules: [
            {
                Status: 'Enabled',
                ID: 'abortMulti',
                Expiration: {ExpiredObjectDeleteMarker: true},
                Filter: {} // <- change is here
            }
        ]
    }
});

✅ Correct

const command = new PutBucketLifecycleConfigurationCommand({
    Bucket: 'mybucket123',
    LifecycleConfiguration: {
        Rules: [
            {
                Status: 'Enabled',
                ID: 'abortMulti',
                Expiration: {ExpiredObjectDeleteMarker: true},
                Filter: {Prefix: {}} // <- change is here 
            }
        ]
    }
});

==========

Golang Example:

❌ Wont Compile:

		LifecycleConfiguration: &types.BucketLifecycleConfiguration{
			Rules: []types.LifecycleRule{
				{
					Status: types.ExpirationStatusEnabled,
					Filter: &types.LifecycleRuleFilter{}, // <- this is an interface not a concrete type. 
					AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: 1,
					},
					Expiration: &types.LifecycleExpiration{
						ExpiredObjectDeleteMarker: true,
					},
					ID: aws.String("abortMulti"),
				},
			},
		},
	})

❌ Incorrect:

		LifecycleConfiguration: &types.BucketLifecycleConfiguration{
			Rules: []types.LifecycleRule{
				{
					Status: types.ExpirationStatusEnabled,
					            // <- missing Filter
					AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: 1,
					},
					Expiration: &types.LifecycleExpiration{
						ExpiredObjectDeleteMarker: true,
					},
					ID: aws.String("abortMulti"),
				},
			},
		},

✅ Correct:

LifecycleConfiguration: &types.BucketLifecycleConfiguration{
			Rules: []types.LifecycleRule{
				{
					Status: types.ExpirationStatusEnabled,
					Filter: &types.LifecycleRuleFilterMemberPrefix{},
					AbortIncompleteMultipartUpload: &types.AbortIncompleteMultipartUpload{
						DaysAfterInitiation: 1,
					},
					Expiration: &types.LifecycleExpiration{
						ExpiredObjectDeleteMarker: true,
					},
					ID: aws.String("abortMulti"),
				},
			},
		},

Again quoting documentation:

• Filter is required if the LifecycleRule does not contain a Prefix element.
• A Filter must have exactly one of Prefix, Tag, or And specified.

The only thing I could argue is that the s3 exception is not detailed enough and requires you to dig through documentation to see what is missing in the XML that is being serialized.

Conclusion: Not a bug.

I will convert this into a discussion and keep it open so I may address more questions if you have them in the future.

Thanks again! Ran 😄