Move dependencies on NIO/AsyncHTTPClient to URLSession

[cmcgee1024]

In an effort to minimize dependencies, it could be that URLSession can be used instead of depending on the AHC/NIO stack. Let's investigate what if anything is missing from URLSession at this point.

[0xTim]

This would require curl to be installed on target machines (which we can't guarantee) and IIRC we can't statically link it into the binary

[gregcotten]

This would pave the way to getting swiftly on Windows, since NIO on Windows seems impossible at the moment.

[jakepetroules]

@0xTim Are you sure? I thought Foundation statically links curl at build time. I don't think there is any expectation that curl needs to be installed on the target machine.

[MaxDesiatov]

I thought Foundation statically links curl at build time.

On Linux it does not.

I don't think there is any expectation that curl needs to be installed on the target machine.

There is an expectation, all of our Linux distribution tarballs require libcurl and its dependencies to be installed, you can see all Docker images doing this as a prerequisite for installation https://github.com/search?q=repo%3Aswiftlang%2Fswift-docker%20libcurl&type=code

I don't think replacing NIO/AHC with URLSession on Linux would minimize dependencies. In fact it would add a ton of new dependencies we can't control since libcurl pulls in cryptography libraries, spread around through dozens of packages on some Linux distributions.

[jakepetroules]

Ah, so it's a Linux-specific issue - good to know, thanks!

Perhaps we could consider using the URLSession backend on Windows and macOS, and retain the NIO one on other platforms, then.

This would pave the way to getting swiftly on Windows, since NIO on Windows seems impossible at the moment.

FWIW, @lukasa says that Windows support for NIO is coming in the next few months.

[0xTim]

Maybe taking a curl dependency on Linux isn't so bad after all, and is actually a better tradeoff vs having a dependency on the large NIO stack. IMO there's pluses and minuses with either path.

The issue with this is that we can't guarantee it will be available on a users machine. So imagine the scenario where a user downloads swiftly from the browser on Linux or using wget. If they then try and run swiftly init it will fail and we can't control the error message, and even if we could it wouldn't make for a good UX. (Ironically this could have been solved if we still had the original install script).

Having NIO as a dependency is not a massive issue IMO since the compile times etc don't affect end users and don't add too much onto it. Especially compared to how awful URLSession is on Linux (the general guidance in the server world is just don't touch it with a barge pole, though it's probably ok for this use case). Windows is definitely an issue however and using URLSession on Windows and AHC on Linux would be a good compromise. (Does URLSession work reliably on Windows?)

[weissi]

removed old message

However, I'm always kind of sad to pull in a NIO dependency because of how much code it pulls in and build time it adds

Not sure I quite follow this. Curl + URLSession is way more code than NIO. The only difference is that SwiftPM/xcodebuild aren't able to use pre-compiled NIO artefacts and compiles it over and over again.

With this argument, we'd drop a lot of Swift code and replace it with unsafe C code just because that is available in pre-compiled form from your distribution and our Swift code isn't and can't be (lack of ABI).

My opinion is that we should instead focus on our build tooling story such that we compile SwiftNIO at most once for the whole system. In that world, I really doubt anybody would even make these arguments.

, especially when many things which use it make little use of its main value-add (the non-blocking I/O), and only need the futures (can use Swift Concurrency instead)

Wait, what? Swift Concurrency is unable to do native I/O but practically requires asynchronous I/O (or else Swift Concurrency doesn't make any sense). So whatever you do, you'll need to bring something that can do async I/O. NIO is doing precisely this bit.

Yes, you can write a HTTP client on top of Dispatch (which also, just like NIO brings an event loop for that) but you won't be able to reach the same performance, nor are the implementations of the protocols actually available. Also, this is a lot of work still. FWIW, libcurl also brings its own event loop internally, it's just implemented in C and pre-compiled by your distribution. Are we really arguing that unsafe C code is preferred over safe Swift code?

and only need the futures

NIO's futures are one way an API can be exposed, Swift Concurrency is another. NIOAsyncChannel is from what I know the only way of doing I/O in Swift Concurrency that's compliant with Structured Concurrency. Also it's actually written in Swift, and happens to scale very well for really very very big production systems that see an enormous amount of traffic.

or an HTTP client (can use Foundation URLSession instead). Maybe taking a curl dependency on Linux isn't so bad after all, and is actually a better tradeoff vs having a dependency on the large NIO stack. IMO there's pluses and minuses with either path.

Again, the "large" NIO stack (NIO + HTTP + TLS) is 7.2 MB compiled in release (with debug symbols, no stripping, just swift build -c release for NIOSSLHTTP1Client). curl + dependencies come in at 10 MB and then another 2 MB for FoundationNetworking itself. And curl+FoundationNetworking will only give you HTTP/TLS, NIO will give you a huge amount of networking servers and clients.

This would pave the way to getting swiftly on Windows, since NIO on Windows seems impossible at the moment.

FWIW, @Lukasa says that Windows support for NIO is coming in the next few months.

This is of course a huge problem and I really hope the NIO team will address this very soon.

Perhaps we could consider using the URLSession backend on Windows and macOS, and retain the NIO one on other platforms, then.

This sounds like the right plan for now. I think it's key to compile a static swiftly binary and this is -- by far -- the best path forward which also happens to provide the best fidelity and smallest binary.

This would pave the way to getting swiftly on Windows, since NIO on Windows seems impossible at the moment. FWIW, @Lukasa says that Windows support for NIO is coming in the next few months.

Yes, this is an issue and I really hope the NIO team are sorting that out soon. It's such a shame that we still can't use NIO on Windows and need to even think about this.

In an effort to minimize dependencies, it could be that URLSession can be used instead of depending on the AHC/NIO stack.

I see no benefits and we'd actually be maximising dependencies. We'd require legacy bits of Foundation and also FoundationNetworking plus a static libcurl copy with another static http2 and TLS implementation. That's just not a good place to be.

FWIW, URLSession on non-Darwin also has a large number of severe issues: https://github.com/swiftlang/swift-corelibs-foundation/issues?q=is%3Aissue%20state%3Aopen%20URLSession