Long scans fail on ExpiredToken when using AWS-IRSA

[nierz]

🐞 Bug report

Describe the bug

When using S3 with auth type of AWS-IRSA, the S3 connection is initiated on Scan creation. AWS-IRSA credentials are limited to 1 hour, and thus, when a scanner takes over 1 hour to complete, the lurker fails with ExpiredToken error (error code 400).

The S3 connection is only relevant once the scanner container finishes running, but it is initiated on setup.

Steps To Reproduce

1. Use S3 with authType: aws-irsa in operator values.yaml.
2. Run a scan that takes over an hour
3. See scanner container completing
4. Lurker container fails on ExpiredToken

Expected behavior

Scanner completes after an hour, lurker uploads file successfully, rest of the flow is executed.

System (please complete the following information):

secureCodeBox version: 4.0.1 (couldn't see anything related to it in newer versions' release notes, related code looks identical)

Kubernetes version:

Client Version: v1.27.4
Kustomize Version: v5.0.1
Server Version: v1.28.4-eks-8cb36c9

Docker version:

Docker version 20.10.18, build b40c2f6b5d

Screenshots / Logs

2024/02/01 06:10:12 File upload returned non 2xx status code (400)
2024/02/01 06:10:12 Failed Request:
2024/02/01 06:10:12 HTTP/1.1 400 Bad Request
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 01 Feb 2024 06:10:11 GMT
Server: AmazonS3
X-Amz-Id-2: mt...
X-Amz-Request-Id: HV...

5c8


<?xml version="1.0" encoding="UTF-8"?>

<Error><Code>ExpiredToken</Code><Message>The provided token has expired.</Message><Token-0>IQ..=</Token-0><RequestId>HV...</RequestId><HostId>mtU5...</HostId></Error>
0

2024/02/01 06:10:12 Lurker failed to upload scan result file. File upload returned non 2xx status code (400)

[J12934]

Hi @nierz

Thanks for reporting this. Also impressive that you got it to work at all as AWS IRSA is barely documented on our side as we still consider the support to be experimental in the operator (in large parts due to this limitation)

Do you have any link confirming this 1 hour expiration? My current understanding is that the IRSA tokens are limited to 8 hours and are refreshed one hour before their expiry.

This also comes down to (in the worst case) 1 hour of token validity when the presigned url is created right before the token is refreshed, as the presigned url can't be valid any longer than the token.

Is that what you meant with the expires after one hour? Because if the acutal IRSA token would only be valid for one hour this would be a way bigger problem as the tokens then might only be valid for a couple of minutes in the worst case.

My hope was that we could that we could find out a way to tell the golang minio client to refetch the tokens early. e.g. after 4 hours to ensure that the is enough time for the scans. I'd personally not want to change the time when the presigned urls are created, I don't see a easy way to fit this into the architecture which would still be easy & secure to run.

[nierz]

Hi @J12934, Thanks for your quick, detailed reply!

I might have mixed the 1 hour limit a bit - looks like that's true for GetSessionToken specifically.

As for the token caching / refreshing - isn't that only true if the remaining token duration is lower than theExpiryWindow, on the NEXT request made? That is, when the initial presigned URL is created, that specific URL will only be valid for the minimum between the duration set for it and the token's duration - assuming no additional requests are issued that would make this token refresh.

As a quick workaround - do presigned URLs created when using credentials for S3 (access-secret-key auth type, rather than aws-irsa) last the full 12 hours? The errors I mentioned happened on a scan that took 1 hour and 10 mins.

Appreciate the help.