Feature Request: How to complete cross-VPC access through App Mesh.

[kevinten10]

Hi, Team.

I'm using AWS App Mesh and I want to inquire about the feasibility of a use case and how to get there.

Which integration(s) is this request for?

Allow me to introduce the requirements:

I have two VPCs that are isolated from each other (but I can communicate via PrivateLink) and I want to be able to call services in the Other VPC from my AWS VPC.

But I don't want the application code to be written differently. (For example, I would like to be able to make calls using an app mesh formatted url, no matter which VPC the service is in).

As shown in the figure, if Service A calls Service C, there is no problem at all, and the call is made through the app mesh using http.

If Service A tries to access Service B in the Other VPC, I want to create a virtual service Server B' in the AWS VPC 's app mesh and redirect the traffic received by Server B' to the Other VPC 's Gateway (maybe by way of Private Link , or forwarding based on other http domain names).

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

In order to solve the above problems, I have two ideas:

Idea 1:

1. Create a virtual service Service B' of Service B in AWS VPC
2. Through the configuration of the app mesh, the traffic received by the virtual service Service B' is forwarded to the Private Link of the Other VPC to realize traffic forwarding.

expected result:

Service A calls Service B' through the app mesh, and the traffic will be forwarded through Private Link.

Idea 2:

I need to write a ServiceProxy myself, its role is to receive all the traffic that needs to cross VPC and forward it to the Gateway of the Other VPC.

Next:

1. Publish Service Proxy to App Mesh
2. Create a virtual service Service B' of Service B in AWS VPC
3. Point the backend of Service B' to the virtual node of Service Proxy

expected result:

Service A calls Service B' through the app mesh, and the traffic is forwarded to the virtual node of Service Proxy. Then Service Proxy completes the forwarding through Private Link.

Are you currently working around this issue?

Yes, I'm trying both of the above ideas. but not yet effective.

So I would like to ask if the solution is feasible and if there is a best practice? If there is no problem with the idea, can you ask me how to configure it?

Thank you so much.

[Y0Username]

@kevinten10 Are these VPCs in different accounts? What are you using for service discovery in the mesh?

If I understand the problem correctly:

• You can use subnet sharing through RAM (if different accounts)
• You can associate these VPCs with same private hosted zones in route53 so that these services are discoverable
• You could also create an endpoint service in other VPC and add an VPCE to it in the primary VPC, Envoys in the Service A, should be able to discover Service B

App Mesh should be able to route traffic if the underlying endpoints are discoverable. please @ me if you need further assistance.

[mahlatsem]

@Y0Username - not related @kevinten10 but have been working on related problem

In our scenario, we already have vpc peering via Transit Gateway and subnet sharing via RAM works The concern with subnet sharing is some rogue auto-scalling event consuming IP space and affecting all the teams that are part of the mesh. The ideal solution would be to keep each team in it's own VPC so we can better manage the blast radius.

Coincidentally, we recently tried associating multiple VPCs to same private hosted zone which works fine with Cloudmap if the VPCs are in the same account, but falls apart when working with multiple accounts because the association does not extend to Cloudmap

If there's another way we can do cross account dns resolution for the ecs services, would love to hear. At this point, seems we have no choice but to use the subnet share and introduce a way to set an upper bound on auto scalling

[kevinten10]

@Y0Username Hi, thanks for your reply.

After trying for a while, I found a better solution here:

• https://docs.aws.amazon.com/app-mesh/latest/userguide/troubleshooting-connectivity.html
• https://github.com/aws/aws-app-mesh-roadmap/issues/65

1. By creating a virtual node A' and pointing its DNS hostname to privatelink (eg www.amazon.com), privatelink points to the gateway under another account.
2. Then create a virtual service A, bind to this virtual node A', and name the virtual service's hostname such as hello-12345.svc.cluster.local
3. Add the /etc/hosts entry to the B APP in the POD corresponding to the virtual node B' bound by the virtual service B, add an /etc/hosts entry (for example, 10.10.10.10 hello-12345), and specify the virtual service A as the backend of the virtual node B'
4. From the B APP, call http://hello-12345.svc.cluster.local, it seems that the request can be forwarded to privatelink (eg www.amazon.com)

After setting it up as above, it seems to have solved my problem so far.

If there are problems with subsequent verifications, I will contact the team again.

Thanks.

[tnsardesai]

@kevinten10 I had a question about your setup if you don't mind. We have come up with a similar setup with 1 difference of transit gateways instead of privatelink as our vpcs are in the same account but different regions.

Our current setup works but intermittently we are seeing some connection issues because of connection resets. I am curious to know what your settings are on the virtual serivce/router/node A' in relation to timeouts and DNS response type. Also curious what type of gateway are you using. If you are using app mesh virtual gateway then is that using an NLB?

@Y0Username I opened https://github.com/aws/aws-app-mesh-roadmap/issues/460 and would love any advice you have