envoy-mobile icon indicating copy to clipboard operation
envoy-mobile copied to clipboard

transport: native sockets on iOS/Android

Open junr03 opened this issue 5 years ago • 11 comments

Using native sockets on iOS/Android is the recommended approach for performing low-level networking. We should investigate:

  • What benefits we can get from using native sockets/APIs
  • The work required to use these APIs, both within Envoy Mobile and upstream Envoy

junr03 avatar Apr 11 '19 17:04 junr03

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

stale[bot] avatar Jun 02 '19 05:06 stale[bot]

Current proposal:

  • Envoy uses BSD sockets for sending/receiving data. On iOS, there are known issues that arise with this approach (such as these with gRPC). To work around these problems, Envoy will need to support using the native networking stack to send/receive data in place of the built-in mechanism
  • Note: We’ll need to take into account TLS support (we should likely use TLS from Envoy instead of native)
  • Implement a socket provider on iOS that will replace the existing BSD socket usage in Envoy. This will need further investigation, but can likely be accomplished through CFStream, Network.framework, etc.

junr03 avatar Jun 17 '19 22:06 junr03

Potentially related, here are some key announcements Apple made regarding networking at WWDC earlier this month: https://github.com/Blackjacx/WWDC#advances-in-networking-part-1 (parts 1 and 2)

rebello95 avatar Jun 18 '19 15:06 rebello95

You'll definitely want to use Network.framework as the socket provider on iOS – it works across all of Apple's platforms. BSD sockets do not work on Apple Watch, for example.

conradev avatar Jun 18 '19 16:06 conradev

an even simpler test: iirc, BSD sockets don't handle cellular/wifi transitions, but Open Transport and Network.framework do. Other areas to check might be multipath TCP support and VPNs.

sayrer avatar Jun 20 '19 16:06 sayrer

For watchers the main work items here are:

  1. Slightly refactor the TLS transport socket to allow for swapping out the actual read/write/connection APIs
  2. Create a new iOS transport socket that derives from the upstream TLS transport socket and uses appropriate iOS APIs for the underlying connections.

This should be pretty straightforward given existing Envoy extensibility mechanisms.

mattklein123 avatar Jun 20 '19 16:06 mattklein123

Detailed write-up of the current state of the transport layer issues on iOS in this comment: https://github.com/lyft/envoy-mobile/issues/128#issuecomment-516260951

rebello95 avatar Jul 30 '19 05:07 rebello95

Current proposal doc.

junr03 avatar Sep 06 '19 17:09 junr03

Related: https://github.com/lyft/envoy-mobile/issues/541

rebello95 avatar Oct 24 '19 19:10 rebello95

https://docs.google.com/document/d/16jq5YO542BbAnBGwldgPfaJel9H3K6ufWD7Lqvrx0Bg/edit?usp=sharing

The above doc captures the current state of the world with copious linked references and examples. In brief, our current usage of POSIX sockets on both platforms, with our current workarounds, presents no serious general-case issues. There are some somewhat niche cases (e.g. on-demand VPN on iOS) which could be better addressed.

In the medium term, implementing socket providers using the newer platform APIs would yield some incremental benefit, but more importantly, help to future-proof the library.

goaway avatar Jul 29 '20 19:07 goaway

While there are no general issues, Multipath TCP could yield pretty significant gains:

Last year we announced that in addition to Siri, we'd also enabled Multipath TCP for Apple Music. Since then, we've seen a 13 percent reduction in music stalls and in the event that a stall occurs, there has been a 22 percent reduction in the stall duration. https://developer.apple.com/videos/play/wwdc2020/10111/

(It might even be more pronounced for Lyft because users are leaving/entering WiFi range all the time)

conradev avatar Aug 01 '20 00:08 conradev