feat: Add TLS support to gnet

[0-haha]

1. Are you opening this pull request for bug-fixes, optimizations or new feature?

new feature

2. Please describe how these code changes achieve your intention.

This PR is to add the TLS support to get.

Change of the source code

• The main TLS library is packaged in the directory pkg/tls/
• internal/boring is the dummy package for boring TLS as it is required by the standard Golang TLS library
• Other changes go to acceptor.go, connection.go, and eventloop.go. Basically, once the TLS enabled on the server side, it will call gnetConn.UpgradeTLS() to upgrade the protocol to TLS. Then, all reads and writes will go to the gnetConn.readTLS() and gnetConn.writeTLS()

The gnet TLS implementation

• is developed based on https://github.com/luyu6056/tls.
• merges the upstream go v1.20rc3 standard TLS library.
  • Since go 1.20 uses crypto/ecdh in crypto/tls/key_agreement.go, which is not available in go <= 1.19, go.mod is bumped up to 1.20.
• adds the Kernel TLS support. So one can offload the encryption to the kernel.
  • The implementation of kernel TLS was based on @jim3m's implementation, which was based on @FiloSottile's implementation.
  • Kernel TLS is totally depending on the kernel version.
  • Kernel TLS Features
    • KTLS 1.2 TX & RX
    • KTLS 1.3 TX & RX
    • zerocopy and no pad for TLS 1.3
    • ciphersuites: AES-GCM-128, AES-GCM-256, CHACHA20POLY1305
  • Kernel TLS TODO
    • KTLS 1.3 RX disabled on kernel < 5.19 as it causes weird package lost
    • zero copy and no pad have not been tested yet. zero copy is enabled on kernel >= 5.19, and no pad is enabled on kernel >= 6.0.
    • sendfile api

Examples

An example of using gnet TLS can found at https://github.com/0-haha/gnet_tls_examples. You should be able to run the repo in docker.

Other Comments

I would say the majority of the implementation has been completed. Open to ongoing conversations. 中文可以聊。

3. Please link to the relevant issues (if any).

Fixes #16

4. Which documentation changes (if any) need to be made/updated because of this PR?

The gnet.Run command will accept the tls.Config like this:

cer, _ := tls.LoadX509KeyPair("server.crt", "server.key")
// server only uses TLS 1.2 and TLS 1.3
config := &tls.Config{
    MinVersion:   tls.VersionTLS12,
    Certificates: []tls.Certificate{cer},
}
gnet.Run(echo, "tcp://192.168.0.100:8000", gnet.WithTLS(config))

4. Checklist

• [ x] I have squashed all insignificant commits.
• [ x] I have commented my code for explaining package types, values, functions, and non-obvious lines.
• [ ] I have written unit tests and verified that all tests passes (if needed).
• [ ] I have documented feature info on the README (only when this PR is adding a new feature).
• [ x] (optional) I am willing to help maintain this change if there are issues with it later.

[panjf2000]

Thank you for implementing TLS and opening this PR.

This might cost me a lot of time to absorb and review the code, but I'll do this as fast as possible.

[0-haha]

I optimized the memory copy and buffer usage for TLS read, write, and handshake. The following briefly describes the implementation idea which would be helpful to review the code

Buffers used in TLS

• rawInput: stores the TLS record from TCP
• input: stores the decrypted TLS record, and the memory is owned by rawInput
• hand: stores the handshake data
• sendBuf: stores the sending data, and it is only used during the handshake

TLS handshake (starts in eventloop.read()):

1. Attach eventloop.buffer to tlsconn.rawInput (zero-copy)
2. Call the tlsconn.handshake()
3. If tlsconn.HandshakeComplete(), call eventloop.readTLS(); Otherwise, return and will restart at 1 in the next round

Note: In tlsconn.handshake(), it extracts the handshake message from tlsconn.rawInput and zero-copys the message to tls.hand. tls.hand discards the messages immediately after using it.

TLS read:

1. Attach eventloop.buffer to tlsconn.rawInput (zero-copy)
2. Call eventloop.readTLS(). Since tlsconn.rawInput can holds multiple TLS records, we iteratively process all TLS records.

   for {
       Extract the TLS record from "tlsconn.rawInput"
       Decrypt the record into "tlsconn.rawInput"
       tlsconn.data = decrypted record (store the reference, zero-copy)
       gnetConn.buffer = tlsconn.Data() (return tlsconn.data, zero-copy)
       eventHandler.OnTraffic()
       c.inboundBuffer.Write(c.buffer)
       if no more TLS records in "tlsconn.rawInput" {
           discard the data in "tlsconn.rawInput" and "tlsconn.data"
           if there is data left in "tlsconn.rawInput", we cache it. so, the left data is not owned by "eventloop.buffer" which will be used by another "gnetConn"
       }
   }
   

   The data pipeline in each iteration is eventloop.buffer (TLS records from TCP) -> (zero-copy) -> tlsconn.rawInput -> (decrypt TLS records) -> tlsconn.rawInput -> (zero-copy) -> tlsconn.Data -> (zero-copy) -> gnetConn.buffer -> (used by eventHandler.OnTraffic()) -> (write rest into to) -> c.inboundBuffer

• Kernel TLS read pipeline eventloop.buffer -> (attach to, zero-copy) -> tlsconn.rawInput ktlsReadRecord -> (decrypt) -> tlsconn.rawInput (referring to eventloop.buffer) -> (zero-copy) -> tlsconn.Data

  The rest follow the standard gnet TLS read.

TLS write:

The data pipeline is data -> (encrypt) -> buffer from sync.Pool -> (call) -> gnetConn.writeTCP() -> (call) -> gnetConn.write() (standard gnet write function) -> return buffer tosync.Pool

• Kernel TLS write call gnetConn.write(data). When gnetConn writes the data into the socket (call unix.Write()), kernel encrypts the data automatically.

[0-haha]

Merge Go upstream commits

[mostafa]

Hey @0-haha @panjf2000,

If you ever need help testing this, I'd be very happy to help.

[panjf2000]

Hey @0-haha @panjf2000,

If you ever need help testing this, I'd be very happy to help.

Thank you for offering to help, I'll reach out to you in case needed.

[panjf2000]

A new thought about TLS in gnet: is it possible to implement TLS as an external library that we are able to plug into gnet (or offload it from gnet) easily?

[0-haha]

A new thought about TLS in gnet: is it possible to implement TLS as an external library that we are able to plug into gnet (or offload it from gnet) easily?

Actually, I have the same idea, like what the quic-go library did. See https://github.com/quic-go/quic-go/tree/master/internal/qtls

Then, it can support multiple go versions at the same time.

One approach is to move the current TLS implementation to a separate repo specifically for go 1.20, and leave the gnet supporting code in this PR. In the future, we simply follow the same approach as quic-go is doing.

[0-haha]

@panjf2000 The TLS implementation has become an external library. gnet can support multiple TLS versions (each binding to a Go version).

See if you are happy with that.

[panjf2000]

@panjf2000 The TLS implementation has become an external library. gnet can support multiple TLS versions (each binding to a Go version).

See if you are happy with that.

Thank you so much for the efforts, I'll review this PR in the next few days.

[Fgaoxing]

@panjf2000 请问gnet何时可以支持TLS