cloudfront update with lambda function for redirects

[crazy-max]

follow-up and take 2 of https://github.com/docker/docker.github.io/pull/15118

As replace directive for an s3 bucket is limited to 50 we can't use them. It's also really slow to propagate each of them with the AWS CLI (~30m).

This pr adds a new function to releaser that will update a lambda edge function to handle redirects directly in our cloudfront distribution.

I have created a lab branch on my fork and also a S3 bucket with a Lambda Edge function attached to a new Cloudfront distribution on my personal AWS account to test this feature and so far so good:

$ curl -s -D - https://dom3zg2ev0we5.cloudfront.net/get-started/part2/ -o /dev/null
HTTP/2 301 
content-length: 0
server: CloudFront
date: Mon, 18 Jul 2022 20:15:47 GMT
location: /get-started/02_our_app/
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 d0229dbe69f77738f3ccab386a045ad8.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG52-P2
x-amz-cf-id: _6ds2VEHsPPvW2w8IuT38Gqw8GmPoJCnMx4rN9MiU9OR1S72LUXDTQ==

Also looking at #15133, I have added a new redirects-prefixes.json file that handles redirects with prefixes so we can get rid of replaces in our s3 bucket with hardcoded domain inside it and have everything in place in our lambda edge function:

$ curl -s -D - https://dom3zg2ev0we5.cloudfront.net/ee/sqdsqdsqdsqds -o /dev/null
HTTP/2 301 
content-length: 0
server: CloudFront
date: Mon, 18 Jul 2022 20:16:33 GMT
location: /
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 8b20ff9a1799265d378bf510ac3db6de.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG52-P2
x-amz-cf-id: KyHvFjoVvM9doj9iuCuAv4CYaxNoIG2GlhhB7KTwEcm2ntQLKDsVqQ==

• Cloudfront URL: https://dom3zg2ev0we5.cloudfront.net/
• Deployment demo: https://github.com/crazy-max/docker.github.io/runs/7397172669?check_suite_focus=true

#16 0.271 INFO: updating lambda function "DockerDocsRedirectFunction"
#16 0.374 INFO: lambda function updated successfully (arn:aws:lambda:us-east-1:707923364795:function:DockerDocsRedirectFunction)
#16 0.374 INFO: waiting for lambda function to be processed
#16 3.549 INFO: lambda function version "2022-07-18-20:12:27" published successfully (arn:aws:lambda:us-east-1:707923364795:function:DockerDocsRedirectFunction:8)
#16 3.701 INFO: cloudfront distribution "E2Q9X128R7SWCF" loaded
#16 3.749 INFO: cloudfront distribution configuration loaded
#16 3.749 INFO: cloudfront distribution viewer request function ARN found: "arn:aws:lambda:us-east-1:707923364795:function:DockerDocsRedirectFunction:7"
#16 3.749 INFO: updating cloudfront config with viewer request function ARN "arn:aws:lambda:us-east-1:707923364795:function:DockerDocsRedirectFunction:8"
#16 5.342 INFO: cloudfront config updated successfully
#16 DONE 5.6s

Keeping in draft for now as we have to create the Lambda Edge function and additional perms on AWS as well as getting the current cloudfront ID for each environment (cc @StefanScherer).

Signed-off-by: CrazyMax [email protected]

[netlify[bot]]

✅ Deploy Preview for docsdocker ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	1b9349357173fcdaebb784a5e4b5784e6164225e
🔍 Latest deploy log	https://app.netlify.com/sites/docsdocker/deploys/62fcd54b0aa8c300091ad95b
😎 Deploy Preview	https://deploy-preview-15151--docsdocker.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[crazy-max]

Labs environment is ready and is using Cloudfront and Lambda Edge to handle redirections. Thanks @VictorBersy!!

$ curl -s -D - https://docs-labs.docker.com/get-started/part2/ -o /dev/null
HTTP/2 301 
content-length: 0
server: CloudFront
date: Mon, 08 Aug 2022 15:08:46 GMT
location: /get-started/02_our_app/
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 f41c2361062c4fc74c645f4e4fddd2de.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG3-C2
x-amz-cf-id: MpXwuPV4N_Kd8sVy_AGNc2Rbi2oniKOVRMDUrBsTFIcbxtiH4Sunbw==

$ curl -s -D - https://docs-labs.docker.com/ee/sqdsqdsqdsqds -o /dev/null
HTTP/2 301 
content-length: 0
server: CloudFront
date: Mon, 08 Aug 2022 15:10:30 GMT
location: /
x-cache: LambdaGeneratedResponse from cloudfront
via: 1.1 9891f2220bf61a27cb1f26085ab3703c.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG3-C2
x-amz-cf-id: Fl9PI6vyWYRIy5kjHpjOEbtJI1kTCf7Onenhtm9wsXxN2_JrFuOQRw==

I will update the PR to reflect changes made on lab branch.

[crazy-max]

Ready for review. I have just enabled OIDC auth and cloudfront redirects with lambda edge func for our lab environment as it's the only one supporting it atm.

Next step I will open a PR to prepare the work for stage and prod environment so that we will be ready when @VictorBersy has created the new stage and prod environments that support OIDC auth and Cloudfront with Lambda Edge redirections as well.

@VictorBersy Let me know when we can catch up on this by giving me the IAM role ARN for OIDC auth and cloudfront id with lambda edge function name as well to use for stage and prod.

cc @thaJeztah @usha-mandya

[AlyxPractice]

We finally managed to make it work with Terraform on the lab branch. Next week I will be working on deploying this to stage. I will create all brand new cloud resources, and make the switch through DNS, to avoid any downtime on stage and to learn how it behaves before hitting production.

Thanks a lot @crazy-max for your support and help 🙇

[crazy-max]

As discussed with @VictorBersy, I have created a dedicated workflow called deploy that will use the new way to deploy our docs with Cloudfront and Lambda Edge so we don't disturb the current one (publish) while testing migration for stage and prod environments.

[crazy-max]

Perfect thanks @VictorBersy! Looks like we are ready to merge. Can you PTAL @thaJeztah? Thanks

[thaJeztah]

Oh! Take note that GitHub is hiding some comments (so may need a "click to load more comments")

[crazy-max]

Should be good, thanks for the review @thaJeztah

[thaJeztah]

okay; let's get this one in.

@VictorBersy for a follow-up; perhaps you have an answer on https://github.com/docker/docker.github.io/pull/15151#discussion_r947721015

Probably too late now (?) to rename these buckets but we should really try to get rid of that docker.github.io everywhere, and use docs.docker.com (also rename the repository back to docs or documentation or docs.docker.com at some point)

I guess it's to match the name of the repository. Am I right @VictorBersy?

[crazy-max]

So far so good for stage2: https://github.com/docker/docker.github.io/runs/7880300358?check_suite_focus=true

#15 0.396 INFO: updating lambda function "DockerDocsRedirectFunction-stage"
#15 0.583 INFO: lambda function updated successfully (arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-stage)
#15 0.584 INFO: waiting for lambda function to be processed
#15 3.824 INFO: lambda function version "2022-08-17T14:05:57Z" published successfully (arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-stage:2)
#15 4.181 INFO: cloudfront distribution "E1R7CSW3F0X4H8" loaded
#15 4.259 INFO: cloudfront distribution configuration loaded
#15 4.259 INFO: cloudfront distribution viewer request function ARN found: "arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-stage:1"
#15 4.259 INFO: updating cloudfront config with viewer request function ARN "arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-stage:2"
#15 6.175 INFO: cloudfront config updated successfully

See https://docs-stage2.docker.com

[crazy-max]

docs2 has been deployed too: https://github.com/docker/docker.github.io/runs/7901183427?check_suite_focus=true#step:9:229

#15 0.475 INFO: updating lambda function "DockerDocsRedirectFunction-prod"
#15 0.707 INFO: lambda function updated successfully (arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-prod)
#15 0.707 INFO: waiting for lambda function to be processed
#15 3.933 INFO: lambda function version "2022-08-18T14:51:50Z" published successfully (arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-prod:2)
#15 4.272 INFO: cloudfront distribution "E228TTN20HNU8F" loaded
#15 4.353 INFO: cloudfront distribution configuration loaded
#15 4.353 INFO: cloudfront distribution viewer request function ARN found: "arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-prod:1"
#15 4.353 INFO: updating cloudfront config with viewer request function ARN "arn:aws:lambda:us-east-1:***:function:DockerDocsRedirectFunction-prod:2"
#15 5.333 INFO: cloudfront config updated successfully
#15 DONE 5.9s

See https://docs2.docker.com/

[thaJeztah]

ooh, and now I'm wondering if we could optimize things further;

curl -v -HEAD https://docs2.docker.com/go/linux-credentials
 TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 302
< content-type: text/html; charset=utf-8
< content-length: 313
< x-amz-error-code: Found
< x-amz-error-message: Resource Found
< x-amz-request-id: QE9X3EZJDS08QN4E
< x-amz-id-2: +YIPXP6llntxLYjqdeVgHk0AJUYqJuNN2gdn5ZNRuzn50KnGsIx3uroaQG5DqM80/WRZJxiODhE=
< location: /go/linux-credentials/
< date: Thu, 18 Aug 2022 15:01:06 GMT
< server: AmazonS3
< x-cache: Hit from cloudfront
< via: 1.1 bdbb0d922c29917c00cfed799f55e7c2.cloudfront.net (CloudFront)
< x-amz-cf-pop: AMS1-C1
< x-amz-cf-id: XBv-kKOMyKAF6x0h0rC2Qrxp5M2WipYWJtINYMU8AaZOjSamtQ7UAQ==
< age: 170
<
<html>
<head><title>302 Moved Temporarily</title></head>
<body>
<h1>302 Moved Temporarily</h1>
<ul>
<li>Code: Found</li>
<li>Message: Resource Found</li>
<li>RequestId: QE9X3EZJDS08QN4E</li>
<li>HostId: +YIPXP6llntxLYjqdeVgHk0AJUYqJuNN2gdn5ZNRuzn50KnGsIx3uroaQG5DqM80/WRZJxiODhE=</li>
</ul>
<hr/>
</body>
</html>

That's a (default?) redirect from s3 to redirect URLs with no trailing /

However, in case of the above URL, that means a double redirect;

curl -v -HEAD https://docs2.docker.com/go/linux-credentials/
..
< location: /desktop/get-started/#credentials-management-for-linux-users

So

1. https://docs2.docker.com/go/linux-credentials -> https://docs2.docker.com/go/linux-credentials/
2. https://docs2.docker.com/go/linux-credentials/ -> https://docs2.docker.com/desktop/get-started/#credentials-management-for-linux-users

Would be cool if our redirect function would be able to skip the first one.

[crazy-max]

That's a (default?) redirect from s3 to redirect URLs with no trailing /

Yes indeed it seems to be returned by s3 and not cloudfront so cannot be handled by our redirect function:

curl -s -D - https://docs2.docker.com/go/linux-credentials -o /dev/null
HTTP/2 302 
content-type: text/html; charset=utf-8
content-length: 313
x-amz-error-code: Found
x-amz-error-message: Resource Found
x-amz-request-id: YHM9DHTDJCZDDW4X
x-amz-id-2: dI4JS/+f1y6O9SVsnLQ/OkK2TFp486XW0B0Ml61RScOj4nHycBLZUrLLg60pEGXhnbir/3XBiiQ=
location: /go/linux-credentials/
date: Thu, 18 Aug 2022 15:19:20 GMT
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 5732b8336788d04c0d6cb18b0b2aa3c2.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG3-C1
x-amz-cf-id: FSbbAeGX8wN0UgLyRE0UmfaQ0XPAZ5EmWqLMDNWsck8MxtRikM5ldg==

See x-cache: Miss from cloudfront.

I guess it needs some tuning on s3 but looks like this is the same behavior currently right?:

curl -s -D - https://docs.docker.com/go/linux-credentials -o /dev/null
HTTP/2 302 
content-type: text/html; charset=utf-8
content-length: 313
x-amz-error-code: Found
x-amz-error-message: Resource Found
location: /go/linux-credentials/
date: Thu, 18 Aug 2022 15:20:32 GMT
server: AmazonS3
x-cache: Miss from cloudfront
via: 1.1 d6bff47a79bb5fa9800d9ee4b2b92146.cloudfront.net (CloudFront)
x-amz-cf-pop: CDG3-C2
x-amz-cf-id: OWQjQSIJRAbTpvzMWyJAsEohk6JWHgkwR0ZBp_u9I0aCP-Y_tZUT-g==

[crazy-max]

Actually I see that we could use our redirect function for this: https://stackoverflow.com/questions/67967925/how-to-redirect-internal-urls-to-end-with-trailing-slashes-with-s3-websites

But I'm afraid we could potentially break access to assets (images, etc..)

[thaJeztah]

But I'm afraid we could potentially break access to assets (images, etc..)

Yes, so I was wondering; it could possibly still work as long as we base our redirects on the https://docs.docker.com/redirects.json; in that case we can look if there's a match for the source URL (after appending a / if there is none) 🤔

The redirects.json doesn't have redirects for assets (at least not currently)

[crazy-max]

Yes, so I was wondering; it could possibly still work as long as we base our redirects on the https://docs.docker.com/redirects.json; in that case we can look if there's a match for the source URL (after appending a / if there is none) 🤔

Hum I see, yes that could work.

[crazy-max]

I changed the redirect function to also check for uri without trailing slash but it doesn't pass through :disappointed:

$ curl -s -D - https://docs-labs.docker.com/go/linux-credentials -o /dev/null
HTTP/2 302 
content-type: text/html; charset=utf-8
content-length: 313
x-amz-error-code: Found
x-amz-error-message: Resource Found
x-amz-request-id: 4FR6HZ1Q4XMGRHBB
x-amz-id-2: ZBCpILJxcFcyWbgFd7g+ZF++/2uWYxphk6KwdOyTi0VWqEHSumUQ7WbnAPv1cBuE9aM5lz+R5Pg=
location: /go/linux-credentials/
date: Thu, 18 Aug 2022 15:57:59 GMT
server: AmazonS3
x-cache: Hit from cloudfront
via: 1.1 c0c888b299b9797c37778648bae22064.cloudfront.net (CloudFront)
x-amz-cf-pop: BRU50-C1
x-amz-cf-id: 3e180Ew3TgspHoj_AaPKG47KFLYsTkMJdIQpWTU9WG5NYsHn-UIoTQ==
age: 17

As I can see from the request, Cloudfront cache that request because it handles itself the redirection without going to the viewer request function. So this is not an S3 config but something in the Cloudfront distribution to change.

As Cloudfront manages this redirection and handles its internal cache for them I think this is fine but we could improve this in a follow-up and let our lambda function do this redirection.

[thaJeztah]

Ah, that's a pity; thanks for trying!