s5cmd icon indicating copy to clipboard operation
s5cmd copied to clipboard

QUIC support

Open jpds opened this issue 2 years ago • 3 comments

I use s5cmd against my own S3-compliant storage which also has QUIC / HTTP/3 support on the frontend traefik servers.

Whilst Amazon S3 doesn't support this, it would be neat if s5cmd could integrate with https://github.com/quic-go/quic-go and upgrade connections to QUIC if support is detected.

jpds avatar Mar 30 '23 16:03 jpds

I've tried implementing this myself, and it appears to be working with just:

diff --git a/storage/s3.go b/storage/s3.go
index 701b525..310d94c 100644
--- a/storage/s3.go
+++ b/storage/s3.go
@@ -29,6 +29,8 @@ import (
        "github.com/aws/aws-sdk-go/service/s3/s3iface"
        "github.com/aws/aws-sdk-go/service/s3/s3manager"
        "github.com/aws/aws-sdk-go/service/s3/s3manager/s3manageriface"
+       "github.com/quic-go/quic-go"
+       "github.com/quic-go/quic-go/http3"

        "github.com/peak/s5cmd/log"
        "github.com/peak/s5cmd/storage/url"
@@ -881,6 +883,16 @@ func (sc *SessionCache) newSession(ctx context.Context, opts Options) (*session.
        }

        var httpClient *http.Client
+
+       var qconf quic.Config
+       roundTripper := &http3.RoundTripper{
+               QuicConfig: &qconf,
+       }
+
+       httpClient = &http.Client{
+               Transport: roundTripper,
+       }
+
        if opts.NoVerifySSL {
                httpClient = insecureHTTPClient
        }

However it does make the client complain about buffer sizes per this upstream document.

I shall experiment with this a bit more.

jpds avatar Mar 30 '23 19:03 jpds

Very good work, please keep us updated once it's implemented. What kind of S3 compliant storage do you use if I may ask?

pr0ton11 avatar May 02 '23 22:05 pr0ton11

@pr0ton11 Garage.

jpds avatar Jul 03 '23 16:07 jpds