tcp: Add TCP_FASTOPEN_CONNECT sockopt

[j47996]

This option causes a subsequent connect syscall to only bind the remote address, leaving the TCP connection initiation for a later data-send call. This supports applications where the transport connect coding is a long way away from the data-send (eg. the latter is in a TLS library) and the wish is for data-bearing TCP Fast Open.

The option name matches that used by the Linux kernel.

For discussion:

• The implementation also does the equivalent of a TCP_FASTOPEN setsockopt; is this wanted or should they be separated? It does not support the PSK mode of TCP_FASTOPEN; is that wanted?
• The implementation diverges the "connected" socket state from the actual state of the TCP connection; is this acceptable architecturally?
• Testing: only kernel bootabilty and feature-exercise testing has been done. Is there a networking testsuite?

[glebius]

In FreeBSD TCP always have had an implied connect. I mean since 1980-ies. That means you can:

s = socket(AF_INET, SOCK_STREAM); sendto(s, data, (struct sockaddr *)&(struct sockaddr_in){.sin_addr = x.x.x.x, sin_port = y})

So, the desired behavior can be achieved at the library level, without any new syscalls. The library can store the sockaddr_in internally and use sendto(2) as the first syscall to initiate the data transfer.

[glebius]

Ah, this also is tied to TFO. I will take a closer look.

[pkelsey]

In FreeBSD TCP always have had an implied connect. I mean since 1980-ies. That means you can:

s = socket(AF_INET, SOCK_STREAM); sendto(s, data, (struct sockaddr *)&(struct sockaddr_in){.sin_addr = x.x.x.x, sin_port = y})

So, the desired behavior can be achieved at the library level, without any new syscalls. The library can store the sockaddr_in internally and use sendto(2) as the first syscall to initiate the data transfer.

@glebius You are correct, the TFO client side support is designed to work in exactly this way. See https://people.freebsd.org/~pkelsey/tfo-tools/tfo-client.c.

[nibanks]

@j47996 It's unclear to me why you need this. Does the existing FreeBSD mechanism not work for you? If you're just trying to match Linux is a client-side shim not a possible option?

[j47996]

@j47996 It's unclear to me why you need this. Does the existing FreeBSD mechanism not work for you? If you're just trying to match Linux is a client-side shim not a possible option?

Because the TLS library code is doing the first write for the connection. This data (the TLS Client Hello) is the data we want to go in the TCP Fast Open SYN packet. I don't get to change that library (which could be any of GnuTLS, OpenSSL, BoringSSL), and I'd rather not have to modify the existing application code (which does the connect() syscall) beyond additional setsockopts at connect-call time or replacing the connect().

As far as I can see, the existing FreeBSD support would require either modification of the TLS library to replace the syscall used for that first data write - or inserting a stateful shim layer underneath that library.

[glebius]

IMHO, what you are trying to achieve is make FreeBSD pretend to be Linux so well, that mentioned libraries won't see any difference and would build on FreeBSD with the same compile options as on Linux. To achieve that you need to fake everything include output of uname(1). This sounds like what Debian/kFreeBSD tried to achieve.

Looking at OpenSSL, file include/internal/bio_tfo.h, I see that the library is written in the standard way how portable libraries are written to use non-portable code. It uses ifdefs, that are either set during the configure stage or are available in OS-specific headers. Particularly OpenSSL will not magically start using TCP_FASTOPEN_CONNECT if this constant is defined on FreeBSD. You would still need to modify the library.

If the goal is to use a library that is not TFO aware at all but force it to use TFO, then I would say that a shim that intercepts connect(2) library call (this lives in libsys in FreeBSD) is a much cleaner solution, than a kernel hack.

[j47996]

Particularly OpenSSL will not magically start using TCP_FASTOPEN_CONNECT if this constant is defined on FreeBSD.

I'm not asking it to.

I'm asking for TCP_FASTOPEN_CONNECT to be available to my application code, where I would use it next to the existing connect() call. The application later goes on to call into the TLS library, which does a write() or sendmsg() without knowing that there might be TFO resulting. That would not have to change.

You are correct that I'm asking for a similar facility to one in Linux. It seems I'm not convincing that it would be useful. If this is so, please close the ticket.

[glebius]

We discussed your submission with a small group of FreeBSD developers, namely tuexen@FreeBSD, rscheff@FreeBSD and couple more guys who aren't committers but have submitted patches to the TCP stack - Peter Lei and Nick Banks. We all agreed that the existing API in FreeBSD is better than the second Linux API (aka TCP_FASTOPEN_CONNECT) came up with. We actually found it easy to adopt the first Linux API of MSG_FASTOPEN flag to sendto(). For the the second Linux API we agreed that it makes sense to hack around a TFO-unaware library, but not a good API for developing a proper TFO application or a lib. And we agreed that such hacks should be better made at library level than at kernel level.

However, our discussion lacked @pkelsey , the principal author and maintainer of TFO in FreeBSD. Let's hear from Patrick, his opinion matters the most.

[tuexen]

I haven't looked at the code, but doesn't support OpenSSL TCP fast open? See NEWS.