Exporting multiple non-Kubernetes services

Open elevran opened this issue 2 years ago • 0 comments

In a demo scenario, we want an app from one Cloud to make use of objects in S3 on AWS. The S3 buckets will be configured so they are only accessible via an AWS VPC, thus, can only be reachable via a ClusterLink gateway deployed in the AWS VPC. The problem I'm having is each S3 bucket has a different endpoint, e.g., [S3 bucket name].s3.amazonaws.com. So, if the app is dealing with 10 buckets, I need to create 10 exports in ClusterLink. If buckets are created and deleted dynamically, e.g., by the app, I would need to also dynamically manage exports accordingly. Would it be possible to support wildcards, e.g., *.s3.amazonaws.com?

This is my findings about S3 hostname to IP address mapping:

I created 3 S3 buckets

mc-ztna-s3-01
mc-ztna-s3-02
mc-ztna-s3-03

The first 2 buckets were created in us-east-2, and the last one was created on us-east-1. It looks like AWS put 8 random load balancers in front of each bucket.

$ dig mc-ztna-s3-01.s3.amazonaws.com

; <<>> DiG 9.10.6 <<>> mc-ztna-s3-01.s3.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35995
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
;; QUESTION SECTION:
;mc-ztna-s3-01.s3.amazonaws.com.	IN	A

;; ANSWER SECTION:
mc-ztna-s3-01.s3.amazonaws.com.	42821 IN CNAME	s3-w.us-east-2.amazonaws.com.
s3-w.us-east-2.amazonaws.com. 2	IN	A	52.219.178.236
s3-w.us-east-2.amazonaws.com. 2	IN	A	52.219.80.172
s3-w.us-east-2.amazonaws.com. 2	IN	A	52.219.84.156
s3-w.us-east-2.amazonaws.com. 2	IN	A	52.219.102.100
s3-w.us-east-2.amazonaws.com. 2	IN	A	52.219.111.4
s3-w.us-east-2.amazonaws.com. 2	IN	A	52.219.176.68
s3-w.us-east-2.amazonaws.com. 2	IN	A	52.219.176.116
s3-w.us-east-2.amazonaws.com. 2	IN	A	52.219.176.148

;; Query time: 40 msec
;; SERVER: 2620:1f7::1#53(2620:1f7::1)
;; WHEN: Thu Sep 14 11:08:37 EDT 2023
;; MSG SIZE  rcvd: 216

$ dig mc-ztna-s3-02.s3.amazonaws.com

; <<>> DiG 9.10.6 <<>> mc-ztna-s3-02.s3.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40233
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
;; QUESTION SECTION:
;mc-ztna-s3-02.s3.amazonaws.com.	IN	A

;; ANSWER SECTION:
mc-ztna-s3-02.s3.amazonaws.com.	42821 IN CNAME	s3-1-w.amazonaws.com.
s3-1-w.amazonaws.com.	26	IN	CNAME	s3-w.us-east-1.amazonaws.com.
s3-w.us-east-1.amazonaws.com. 5	IN	A	16.182.104.185
s3-w.us-east-1.amazonaws.com. 5	IN	A	52.217.236.9
s3-w.us-east-1.amazonaws.com. 5	IN	A	54.231.134.209
s3-w.us-east-1.amazonaws.com. 5	IN	A	54.231.196.81
s3-w.us-east-1.amazonaws.com. 5	IN	A	3.5.17.230
s3-w.us-east-1.amazonaws.com. 5	IN	A	3.5.25.99
s3-w.us-east-1.amazonaws.com. 5	IN	A	3.5.28.205
s3-w.us-east-1.amazonaws.com. 5	IN	A	3.5.29.125

;; Query time: 55 msec
;; SERVER: 2620:1f7::1#53(2620:1f7::1)
;; WHEN: Thu Sep 14 11:08:48 EDT 2023
;; MSG SIZE  rcvd: 237

$ dig mc-ztna-s3-03.s3.amazonaws.com

; <<>> DiG 9.10.6 <<>> mc-ztna-s3-03.s3.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57989
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
;; QUESTION SECTION:
;mc-ztna-s3-03.s3.amazonaws.com.	IN	A

;; ANSWER SECTION:
mc-ztna-s3-03.s3.amazonaws.com.	42821 IN CNAME	s3-1-w.amazonaws.com.
s3-1-w.amazonaws.com.	16	IN	CNAME	s3-w.us-east-1.amazonaws.com.
s3-w.us-east-1.amazonaws.com. 5	IN	A	52.217.32.4
s3-w.us-east-1.amazonaws.com. 5	IN	A	52.217.106.28
s3-w.us-east-1.amazonaws.com. 5	IN	A	52.217.234.169
s3-w.us-east-1.amazonaws.com. 5	IN	A	3.5.2.176
s3-w.us-east-1.amazonaws.com. 5	IN	A	3.5.2.206
s3-w.us-east-1.amazonaws.com. 5	IN	A	3.5.29.188
s3-w.us-east-1.amazonaws.com. 5	IN	A	52.216.43.233
s3-w.us-east-1.amazonaws.com. 5	IN	A	52.216.244.220

;; Query time: 42 msec
;; SERVER: 2620:1f7::1#53(2620:1f7::1)
;; WHEN: Thu Sep 14 11:08:58 EDT 2023
;; MSG SIZE  rcvd: 237

$ dig mc-ztna-s3-03.s3.amazonaws.com

; <<>> DiG 9.10.6 <<>> mc-ztna-s3-03.s3.amazonaws.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2403
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1220
;; QUESTION SECTION:
;mc-ztna-s3-03.s3.amazonaws.com.	IN	A

;; ANSWER SECTION:
mc-ztna-s3-03.s3.amazonaws.com.	42616 IN CNAME	s3-1-w.amazonaws.com.
s3-1-w.amazonaws.com.	133	IN	CNAME	s3-w.us-east-1.amazonaws.com.
s3-w.us-east-1.amazonaws.com. 4	IN	A	3.5.28.104
s3-w.us-east-1.amazonaws.com. 4	IN	A	3.5.29.167
s3-w.us-east-1.amazonaws.com. 4	IN	A	16.182.32.33
s3-w.us-east-1.amazonaws.com. 4	IN	A	52.216.94.219
s3-w.us-east-1.amazonaws.com. 4	IN	A	52.217.163.153
s3-w.us-east-1.amazonaws.com. 4	IN	A	52.217.173.225
s3-w.us-east-1.amazonaws.com. 4	IN	A	54.231.199.249
s3-w.us-east-1.amazonaws.com. 4	IN	A	3.5.25.169

I used dig on mc-ztna-s3-03 twice at the end, and you can see it resolved to a different list of load balancers.

Sep 21 '23 15:09 elevran