parsing error in GetBucketLifecycleConfiguration when <ExpiredObjectDeleteMarker> tag is empty

[prembhaskal]

Describe the bug

The library throws an error in API GetBucketLifecycleConfiguration when the S3 storage returns a empty <ExpiredObjectDeleteMarker> tag.

Expected Behavior

The library should not throw error when tag is <ExpiredObjectDeleteMarker> is empty. for the same output, the AWS S3 CLI is able to properly parse and return the bucket lifecycle.

Current Behavior

Below error is thrown:

$ env NOOBAA_ACCESS_KEY=EoBqQuwbZXDl5mq4l0q7 NOOBAA_SECRET_KEY=hKhDHKexL72z4DP6tABqTrRbNiUKRAZiaB3TFVvu S3_ENDPOINT=http://localhost:6001 go run test_lf_bug.go
SDK 2022/04/30 13:10:48 DEBUG Request
GET /del-marker-test?lifecycle= HTTP/1.1
Host: localhost:6001
User-Agent: aws-sdk-go-v2/1.13.0 os/linux lang/go/1.17.6 md/GOOS/linux md/GOARCH/amd64 api/s3/1.24.0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: d0d89662-e395-40dc-8c47-546e5da5bfe3
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=EoBqQuwbZXDl5mq4l0q7/20220430/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=b526c53055acb2a37316373d72d66c143c2927667b0acaea050a9300c9a8d7f7
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220430T074048Z

SDK 2022/04/30 13:10:48 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 303
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: ETag,X-Amz-Version-Id
Connection: keep-alive
Content-Type: application/xml
Date: Sat, 30 Apr 2022 07:40:48 GMT
Keep-Alive: timeout=5
X-Amz-Id-2: l2lk17br-dmytcc-17z4
X-Amz-Request-Id: l2lk17br-dmytcc-17z4

<?xml version="1.0" encoding="UTF-8"?><LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ID>data-expire</ID><Prefix></Prefix><Status>Enabled</Status><Expiration><Days>1</Days><ExpiredObjectDeleteMarker></ExpiredObjectDeleteMarker></Expiration></Rule></LifecycleConfiguration>
panic: err in get lifecycle: operation error S3: GetBucketLifecycleConfiguration, https response error StatusCode: 200, RequestID: l2lk17br-dmytcc-17z4, HostID: l2lk17br-dmytcc-17z4, deserialization failed, failed to decode response body, expected ExpiredObjectDeleteMarker to be of type *bool, got []uint8 instead

Reproduction Steps

steps to reproduce

• connect to ODF/Noobaa S3 store
• create bucket, enable version
• add a lifecycle rule with delete marker set to false
• confirm using s3api that lifecycle rule is properly created
• run the attached go code to connect to S3 store and get the lifecycle

$ S3_ROUTE=http://localhost:6001
$ NOOBAA_ACCESS_KEY=EoBqQuwbZXDl5mq4l0q7
$ NOOBAA_SECRET_KEY=hKhDHKexL72z4DP6tABqTrRbNiUKRAZiaB3TFVvu
$ alias s3='AWS_ACCESS_KEY_ID=$NOOBAA_ACCESS_KEY AWS_SECRET_ACCESS_KEY=$NOOBAA_SECRET_KEY aws --endpoint $S3_ROUTE --no-verify-ssl s3'
$ alias s3api='AWS_ACCESS_KEY_ID=$NOOBAA_ACCESS_KEY AWS_SECRET_ACCESS_KEY=$NOOBAA_SECRET_KEY aws --endpoint $S3_ROUTE --no-verify-ssl s3api'

$ # create bucket
$ s3 mb s3://del-marker-test

$ s3api put-bucket-versioning --bucket del-marker-test --versioning-configuration Status=Enabled

$ cat > expire-lfrule.json << EOF
{
    "Rules": [
        {
            "Expiration": {
                "Days": 1,
                "ExpiredObjectDeleteMarker": false
            },
            "ID": "data-expire",
            "Prefix": "",
            "Status": "Enabled"
        }
    ]
}
EOF

$ s3api put-bucket-lifecycle-configuration --bucket del-marker-test --lifecycle-configuration file://expire-lfrule.json
$ s3api get-bucket-lifecycle-configuration --bucket del-marker-test
{
    "Rules": [
        {
            "Expiration": {
                "Days": 1,
                "ExpiredObjectDeleteMarker": false
            },
            "ID": "data-expire",
            "Prefix": "",
            "Status": "Enabled"
        }
    ]
}

$ env NOOBAA_ACCESS_KEY=EoBqQuwbZXDl5mq4l0q7 NOOBAA_SECRET_KEY=hKhDHKexL72z4DP6tABqTrRbNiUKRAZiaB3TFVvu S3_ENDPOINT=http://localhost:6001 go run test_lf_bug.go
SDK 2022/04/30 13:10:48 DEBUG Request
GET /del-marker-test?lifecycle= HTTP/1.1
Host: localhost:6001
User-Agent: aws-sdk-go-v2/1.13.0 os/linux lang/go/1.17.6 md/GOOS/linux md/GOARCH/amd64 api/s3/1.24.0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: d0d89662-e395-40dc-8c47-546e5da5bfe3
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=EoBqQuwbZXDl5mq4l0q7/20220430/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=b526c53055acb2a37316373d72d66c143c2927667b0acaea050a9300c9a8d7f7
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220430T074048Z

SDK 2022/04/30 13:10:48 DEBUG Response
HTTP/1.1 200 OK
Content-Length: 303
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256
Access-Control-Allow-Methods: GET,POST,PUT,DELETE,OPTIONS
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: ETag,X-Amz-Version-Id
Connection: keep-alive
Content-Type: application/xml
Date: Sat, 30 Apr 2022 07:40:48 GMT
Keep-Alive: timeout=5
X-Amz-Id-2: l2lk17br-dmytcc-17z4
X-Amz-Request-Id: l2lk17br-dmytcc-17z4

<?xml version="1.0" encoding="UTF-8"?><LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ID>data-expire</ID><Prefix></Prefix><Status>Enabled</Status><Expiration><Days>1</Days><ExpiredObjectDeleteMarker></ExpiredObjectDeleteMarker></Expiration></Rule></LifecycleConfiguration>
panic: err in get lifecycle: operation error S3: GetBucketLifecycleConfiguration, https response error StatusCode: 200, RequestID: l2lk17br-dmytcc-17z4, HostID: l2lk17br-dmytcc-17z4, deserialization failed, failed to decode response body, expected ExpiredObjectDeleteMarker to be of type *bool, got []uint8 instead

goroutine 1 [running]:
main.main()
	/home/prem/go/src/gerrit.ext.net.nokia.com/AANM/sandbox-team3/minio-replace-library/s3v2/aws_lf_bug/test_lf_bug.go:27 +0x128
exit status 2

go code to reproduce issue

package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"net/http"
	"os"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials"
	"github.com/aws/aws-sdk-go-v2/service/s3"
	"github.com/aws/aws-sdk-go-v2/service/s3/types"
)

func main() {
	client, err := getS3Client(true)
	if err != nil {
		panic(fmt.Sprintf("err in create client: %s", err.Error()))
	}

	bucketName := "del-marker-test"
	params := &s3.GetBucketLifecycleConfigurationInput{Bucket: &bucketName}
	lfconfig, err := client.GetBucketLifecycleConfiguration(context.TODO(), params)
	if err != nil {
		panic(fmt.Sprintf("err in get lifecycle: %s", err.Error()))
	}

	// fmt.Printf("bucket lifecycle output: %+v\n\n", lfconfig)
	printBucketLifecycle(lfconfig.Rules)
}

func getS3Client(isAWSDebugEnabled bool) (*s3.Client, error) {
	accesskey := os.Getenv("NOOBAA_ACCESS_KEY")
	secretkey := os.Getenv("NOOBAA_SECRET_KEY")
	endpoint := os.Getenv("S3_ENDPOINT")
	return GetS3ClientWithCred(isAWSDebugEnabled, accesskey, secretkey, endpoint)
}

// GetS3ClientWithCred creats s3 client with given access and secret key.
func GetS3ClientWithCred(isAWSDebugEnabled bool, accesskey, secretkey, endpoint string) (*s3.Client, error) {
	defaultRegion := "us-east-1"
	customCredentialResolver := credentials.NewStaticCredentialsProvider(accesskey, secretkey, "")
	customEndpointResolver := aws.EndpointResolverFunc(func(service, region string) (aws.Endpoint, error) {
		return aws.Endpoint{
			PartitionID:       "aws",
			URL:               endpoint,
			SigningRegion:     defaultRegion,
			HostnameImmutable: true,
		}, nil
	})

	logMode := aws.ClientLogMode(0)
	if isAWSDebugEnabled {
		logMode |= aws.LogRequest | aws.LogResponse | aws.LogResponseWithBody
	}

	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	no_ssl_verify := &http.Client{Transport: tr}

	// config with request logging enabled, see more here  https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/logging/
	cfg, err := config.LoadDefaultConfig(context.TODO(),
		config.WithRegion(defaultRegion),
		config.WithCredentialsProvider(customCredentialResolver),
		config.WithEndpointResolver(customEndpointResolver),
		config.WithClientLogMode(logMode),
		config.WithHTTPClient(no_ssl_verify),
	)
	client := s3.NewFromConfig(cfg)
	return client, err
}

func printBucketLifecycle(rules []types.LifecycleRule) {
	for _, rule := range rules {
		fmt.Printf("rule ID: %s\n", *rule.ID)
		fmt.Printf("Status: %s\n", rule.Status)
		printLifecycleRuleFilter(rule.Filter)
		if exp := rule.Expiration; exp != nil {
			printLifeCycleExpiration(*exp)
		}
		if noncexp := rule.NoncurrentVersionExpiration; noncexp != nil {
			fmt.Printf("NoncurrentExpiration: \n")
			fmt.Printf("  Days: %d\n", noncexp.NoncurrentDays)
		}
		fmt.Println()
	}
}

func printLifeCycleExpiration(lfexp types.LifecycleExpiration) {
	fmt.Printf("Expiration: \n")
	if lfexp.ExpiredObjectDeleteMarker {
		fmt.Printf("  DeleteMarker: true\n")
		return
	}
	if lfexp.Date != nil {
		fmt.Printf("  Date: %v\n", *lfexp.Date)
	}
	fmt.Printf("  Days: %d\n", lfexp.Days)
}

func printLifecycleRuleFilter(filter types.LifecycleRuleFilter) {
	fmt.Printf("Filter: \n")
	switch v := filter.(type) {
	case *types.LifecycleRuleFilterMemberPrefix:
		fmt.Printf("  Prefix: %q\n", v.Value)
	default:
		fmt.Printf("  Prefix: %+v\n", v)
	}
}

Possible Solution

This https://github.com/aws/aws-sdk-go-v2/pull/1681 should solve it, but it was pointed out that I had done changes in generated code.

Additional Information/Context

There can be one argument that the S3 storage (ODF/Noobaa) should fix on their side and not return an empty tag. <ExpiredObjectDeleteMarker></ExpiredObjectDeleteMarker>

I am going to raise a ticket for them too.

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.13.0
github.com/aws/aws-sdk-go-v2/config v1.13.0
github.com/aws/aws-sdk-go-v2/credentials v1.8.0
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.9.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.24.0

Compiler and Version used

go version go1.17.6 linux/amd64

Operating System and version

(virtualbox) Linux prem-lubuntu 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[prembhaskal]

hi, is this reported issue going to be considered ? Are some details needed here?

Thanks, Prem

[RanVaknin]

Hi @prembhaskal ,

How did you create the policy that you are trying to retrieve? The console might let you create an XML policy that does not adhere to the rules specified by the S3 API.

If you create the desired policy through the SDK you should be able to retrieve it. This customer ran into a similar issue with a serialization error.

You can also checkout if this discussion helps at all. Thanks, Ran~

[github-actions[bot]]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.