utls
utls copied to clipboard
Using utls with `http.Transport`
I'm using commit a89e7e6da482a5a0db02578fc606ace9ccfbea62. The examples I have found of using utls with HTTPS all make a single request on a single connection, then throw the connection away. For example, httpGetOverConn in examples.go.
I'm trying to use utls with http.Transport, to take advantage of persistent connections and reasonable default timeouts. To do this, I'm hooking into the DialTLS callback. There is a problem when using a utls fingerprint that includes h2 in ALPN and a server that supports HTTP/2. The server switches to HTTP/2 mode, but the client stays in HTTP/1.1 mode, because net/http disables automatic HTTP/2 support whenever DialTLS is set. The end result is an HTTP/1.1 client speaking to an HTTP/2 server; i.e, a similar problem as what was reported in https://github.com/golang/go/issues/14275#issue-132513041. The error message differs depending on the fingerprint:
HelloFirefox_63net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x00@\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01\x00\x00\x1e\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01http2_handshake_failed"HelloChrome_70local error: tls: unexpected messageHelloIOS_11_12019/01/11 14:48:56 Unsolicited response received on idle HTTP channel starting with "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x00@\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01"; err=<nil>
readLoopPeekFailLocked: <nil>
I get the same results even if I pre-configure the http.Transport with HTTP/2 support by calling http2.ConfigureTransport(tr).
I wrote a test program to reproduce these results. It takes a -utls option to select a utls client hello ID, and a -callhandshake option to control whether to call UConn.Handshake within DialTLS, or allow it to be called implicitly by the next Read or Write. I included the latter option because I found that not calling UConn.Handshake inside DialTLS avoids the HTTP version mismatch; however it also results in a client hello that lacks ALPN and differs from the requested one in other ways, so it's not an adequate workaround.
Click to expand program
package main
import (
"flag"
"fmt"
"io"
"io/ioutil"
"net"
"net/http"
"os"
"strings"
utls "github.com/refraction-networking/utls"
)
func main() {
utlsClientHelloIDName := flag.String("utls", "", "use utls with the given ClientHelloID (e.g. HelloGolang)")
callHandshake := flag.Bool("callhandshake", false, "call UConn.Handshake inside DialTLS")
flag.Parse()
if *callHandshake && *utlsClientHelloIDName == "" {
fmt.Fprintf(os.Stderr, "error: -callhandshake only makes sense with -utls\n")
os.Exit(1)
}
if flag.NArg() != 1 {
fmt.Fprintf(os.Stderr, "error: need a URL\n")
os.Exit(1)
}
url := flag.Arg(0)
utlsClientHelloID, ok := map[string]*utls.ClientHelloID{
"": nil,
"HelloGolang": &utls.HelloGolang,
"HelloRandomized": &utls.HelloRandomized,
"HelloRandomizedALPN": &utls.HelloRandomizedALPN,
"HelloRandomizedNoALPN": &utls.HelloRandomizedNoALPN,
"HelloFirefox_Auto": &utls.HelloFirefox_Auto,
"HelloFirefox_55": &utls.HelloFirefox_55,
"HelloFirefox_56": &utls.HelloFirefox_56,
"HelloFirefox_63": &utls.HelloFirefox_63,
"HelloChrome_Auto": &utls.HelloChrome_Auto,
"HelloChrome_58": &utls.HelloChrome_58,
"HelloChrome_62": &utls.HelloChrome_62,
"HelloChrome_70": &utls.HelloChrome_70,
"HelloIOS_Auto": &utls.HelloIOS_Auto,
"HelloIOS_11_1": &utls.HelloIOS_11_1,
}[*utlsClientHelloIDName]
if !ok {
fmt.Fprintf(os.Stderr, "unknown client hello ID %q\n", *utlsClientHelloIDName)
os.Exit(1)
}
tr := http.DefaultTransport.(*http.Transport)
if utlsClientHelloID != nil {
tr.DialContext = nil
tr.Dial = func(network, addr string) (net.Conn, error) { panic("Dial should not be called") }
tr.DialTLS = func(network, addr string) (net.Conn, error) {
fmt.Printf("DialTLS(%q, %q)\n", network, addr)
if tr.TLSClientConfig != nil {
fmt.Printf("warning: ignoring TLSClientConfig %v\n", tr.TLSClientConfig)
}
conn, err := net.Dial(network, addr)
if err != nil {
return nil, err
}
uconn := utls.UClient(conn, nil, *utlsClientHelloID)
colonPos := strings.LastIndex(addr, ":")
if colonPos == -1 {
colonPos = len(addr)
}
uconn.SetSNI(addr[:colonPos])
if *callHandshake {
err = uconn.Handshake()
}
return uconn, err
}
}
for i := 0; i < 4; i++ {
resp, err := get(tr, url)
if err != nil {
fmt.Printf("%2d err %v\n", i, err)
} else {
fmt.Printf("%2d %s %s\n", i, resp.Proto, resp.Status)
}
}
}
func get(rt http.RoundTripper, url string) (*http.Response, error) {
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
resp, err := rt.RoundTrip(req)
if err != nil {
return nil, err
}
// Read and close the body to enable connection reuse with HTTP/1.1.
_, err = io.Copy(ioutil.Discard, resp.Body)
if err != nil {
return nil, err
}
err = resp.Body.Close()
if err != nil {
return nil, err
}
return resp, nil
}
Sample usage:
test -utls HelloFirefox_63 -callhandshake https://golang.org/robots.txt
The output of the program appears in the following table. Things to notice:
DialTLSwithHelloGolangproduces a fingerprint that is different from usinghttp.TransportwithoutDialTLSset.HelloFirefox_63,HelloChrome_70, andHelloIOS_11_1all provide a usable connection (but with an incorrect fingerprint), as long as you don't callUConn.Handshakebefore returning fromDialTLS.HelloFirefox_63,HelloChrome_70, andHelloIOS_11_1all give the correct fingerprint, but fail with an HTTP version mismatch, whenUConn.Handshakeis called insideDialTLS.
| Client Hello ID | call Handshake? |
client ALPN | result |
|---|---|---|---|
| none | N/A | [h2, http/1.1] | ok HTTP/2 |
-utls HelloGolang |
none | ok HTTP/1.1 | |
-utls HelloGolang |
-callhandshake |
none | ok HTTP/1.1 |
-utls HelloFirefox_63 |
none | ok HTTP/1.1 | |
-utls HelloFirefox_63 |
-callhandshake |
[h2, http/1.1] | malformed HTTP response (HTTP/1.1 client, HTTP/2 server) |
-utls HelloChrome_70 |
none | ok HTTP/1.1 | |
-utls HelloChrome_70 |
-callhandshake |
[h2, http/1.1] | local error: tls: unexpected message |
-utls HelloIOS_11_1 |
none | ok HTTP/1.1 | |
-utls HelloIOS_11_1 |
-callhandshake |
[h2, h2-16, h2-15, h2-14, spdy/3.1, spdy/3, http/1.1] | readLoopPeekFailLocked: <nil> (HTTP/1.1 client, HTTP/2 server) |
Is there a way to accomplish what I am trying to do?
I had some success by switching http2.Transport for http.Transport:
package main
import (
+ gotls "crypto/tls"
"flag"
"fmt"
"io"
@@ -11,6 +12,7 @@ import (
"strings"
utls "github.com/refraction-networking/utls"
+ "golang.org/x/net/http2"
)
func main() {
@@ -51,20 +53,15 @@ func main() {
os.Exit(1)
}
- tr := http.DefaultTransport.(*http.Transport)
+ tr := &http2.Transport{}
if utlsClientHelloID != nil {
- tr.DialContext = nil
- tr.Dial = func(network, addr string) (net.Conn, error) { panic("Dial should not be called") }
- tr.DialTLS = func(network, addr string) (net.Conn, error) {
+ tr.DialTLS = func(network, addr string, cfg *gotls.Config) (net.Conn, error) {
fmt.Printf("DialTLS(%q, %q)\n", network, addr)
- if tr.TLSClientConfig != nil {
- fmt.Printf("warning: ignoring TLSClientConfig %v\n", tr.TLSClientConfig)
- }
conn, err := net.Dial(network, addr)
if err != nil {
return nil, err
}
- uconn := utls.UClient(conn, nil, *utlsClientHelloID)
+ uconn := utls.UClient(conn, &utls.Config{NextProtos: cfg.NextProtos}, *utlsClientHelloID)
colonPos := strings.LastIndex(addr, ":")
if colonPos == -1 {
colonPos = len(addr)
These are the results using the URL https://golang.org/robots.txt. Notice:
- The output is the same whether
UConn.Handshakeis called withinDialTLSor not. - The
HelloChrome_70fingerprint doesn't work.
| Client Hello ID | call Handshake? |
client ALPN | result |
|---|---|---|---|
| none | N/A | [h2] | ok HTTP/2 |
-utls HelloGolang |
[h2] | ok HTTP/2 | |
-utls HelloGolang |
-callhandshake |
[h2] | ok HTTP/2 |
-utls HelloFirefox_63 |
[h2, http/1.1] | ok HTTP/2 | |
-utls HelloFirefox_63 |
-callhandshake |
[h2, http/1.1] | ok HTTP/2 |
-utls HelloChrome_70 |
[h2, http/1.1] | local error: tls: unexpected message |
|
-utls HelloChrome_70 |
-callhandshake |
[h2, http/1.1] | local error: tls: unexpected message |
-utls HelloIOS_11_1 |
[h2, h2-16, h2-15, h2-14, spdy/3.1, spdy/3, http/1.1] | ok HTTP/2 | |
-utls HelloIOS_11_1 |
-callhandshake |
[h2, h2-16, h2-15, h2-14, spdy/3.1, spdy/3, http/1.1] | ok HTTP/2 |
Aside from the HelloChrome_70 failure, this is basically what I want. The only problem is that it doesn't work against non-HTTP/2 servers, whether utls is used or not:
$ test https://apache.org/
0 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
1 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
2 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
3 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
$ test -utls HelloGolang -callhandshake https://apache.org/
DialTLS("tcp", "apache.org:443")
0 err unexpected EOF
DialTLS("tcp", "apache.org:443")
1 err unexpected EOF
DialTLS("tcp", "apache.org:443")
2 err unexpected EOF
DialTLS("tcp", "apache.org:443")
3 err unexpected EOF
Yawning has a commit adding uTLS to obfs4proxy's meek_lite mode:
https://gitlab.com/yawning/obfs4/commit/4d453dab2120082b00bf6e63ab4aaeeda6b8d8a3
It uses an I idea I didn't think of. Instead of setting DialTLS on an http.Transport and using it directly, it uses a wrapper http.RoundTripper. On the first dial, the wrapper initiates the UClient connection, then inspects conn.ConnectionState().NegotiatedProtocol. Depending on the negotiated protocol, it creates internally either an http.Transport or http2.Transport. Future dials are passed through to the internal transport directly.
This is a terrific solution. I'd like to get a wrapper like this into uTLS.
Note that my implementation makes certain assumptions that may not be valid for a more general wrapper (omits some locking, destination host is assumed to be static), but there's comments where I do such things, and altering the behavior should be trivial.
This is a terrific solution. I'd like to get a wrapper like this into uTLS.
Was there ever a proper soloution to use uTLS with the net/http Client?
Comments above describe a proper solution.
This wrapper had not been implemented in uTLS yet.

Note that my implementation makes certain assumptions that may not be valid for a more general wrapper (omits some locking, destination host is assumed to be static), but there's comments where I do such things, and altering the behavior should be trivial.
I'm trying to write a wrapper for net.http Client based on your soloution. Am I right in thinking that I would need to execute getTransport() for every new host and do some kind of connection pooling for effeciency?
EDIT: I was thinking about this incorrectly regarding connection pooling, The only thing required would be to use something like a map[string]net.Conn so we can handle multiple hosts. Everything seems to be working correctly, after some testing ill submit a PR.
the DialTLS says it's for non-proxied HTTPS requests,But if I want to make requests with transport and proxy,How to make that?
the DialTLS says it's for non-proxied HTTPS requests,But if I want to make requests with transport and proxy,How to make that?
There are some code samples here, from the uTLS integration into meek. Adding uTLS was this commit:
Adding proxy support was these commits:
The problem (or at least part of it) appears to be here: https://github.com/golang/go/blob/ea1437a8cdf6bb3c2d2447833a5d06dbd75f7ae4/src/net/http/transport.go#L1496
The http library determines whether to use h2 or not by evaluating the pconn.tlsState field, which gets set here: https://github.com/golang/go/blob/ea1437a8cdf6bb3c2d2447833a5d06dbd75f7ae4/src/net/http/transport.go#L1513
Since it fails to cast the conn to a tls.Conn (since it's a utls.Conn, which implements the same net.Conn interface but cannot be cast to a tls.Conn), it doesn't set the pconn.tlsState field, and therefore doesn't know later on that it's an h2 connection.
What would be ideal is some way to convert a utls.Conn to a tls.Conn (which shouldn't pose any issues once the handshake is already completed, I think?), and return that from the DialTLSContext custom function.
the DialTLS says it's for non-proxied HTTPS requests,But if I want to make requests with transport and proxy,How to make that?
There are some code samples here, from the uTLS integration into meek. Adding uTLS was this commit:
Adding proxy support was these commits:
Very exciting solution.
I try to make some changes to it, because I especially need a react to req.Context.
For example, in httpProxy.Dial transformed into httpProxy.DialContext.
But in the follow-up, I found that by default, http2.Transport is used directly, and it internally initializes http2clientConnPool instead of http2noDialClientConnPool.
So its dial does not use the logic of http.Transport, which causes it to be used in dial Does not support req.Context react.
Is there any way to use the RoundTripper solution with a Client so you can set Redirect handler and proxy?
This comment is just to report a negative result, an idea I had that turned out not to work. Setting ForceAttemptHTTP2 to true on an http.Transport that uses a uTLS dialer does not suffice to negotiate an HTTP/2 session. This is probably for the reason @KyleKotowick noted in https://github.com/refraction-networking/utls/issues/16#issuecomment-624076510.
When this issue was crated, go1.11 was current. go1.13 added Transport.ForceAttemptHTTP2:
The new field
Transport.ForceAttemptHTTP2controls whether HTTP/2 is enabled when a non-zeroDial,DialTLS, orDialContextfunc orTLSClientConfigis provided.
The commits that added this field are https://github.com/golang/go/commit/94e720059f902339699b8bf7b2b10897311b50f8 and https://github.com/golang/go/commit/2a931bad4e4d7eabdb685b6dc74406373ad6e7e4.
The following example program shows that setting ForceAttemptHTTP2 does not work; the client negotiates HTTP/1 even when the server supports HTTP/2. This is not too surprising, since DefaultTransport has ForceAttemptHTTP2: true.
demo.go
// Demonstration that setting net/http Transport.ForceAttemptHTTP2 to true does
// not suffice to enable HTTP/2 support when its dialer returns a *utls.Conn,
// rather than a *tls.Conn.
package main
import (
"context"
"flag"
"fmt"
"net"
"net/http"
"os"
utls "github.com/refraction-networking/utls"
)
var utlsClientHelloID = &utls.HelloFirefox_65
// utlsDialContext connects to the given network address and initiates a TLS
// handshake with the provided ClientHelloID, and returns the resulting TLS
// connection.
func utlsDialContext(ctx context.Context, network, addr string, config *utls.Config, id *utls.ClientHelloID) (*utls.UConn, error) {
// Set the SNI from addr, if not already set.
if config == nil {
config = &utls.Config{}
}
if config.ServerName == "" {
config = config.Clone()
host, _, err := net.SplitHostPort(addr)
if err != nil {
return nil, err
}
config.ServerName = host
}
dialer := &net.Dialer{}
conn, err := dialer.DialContext(ctx, network, addr)
if err != nil {
return nil, err
}
uconn := utls.UClient(conn, config, *id)
// Manually remove the SNI if it contains an IP address.
// https://github.com/refraction-networking/utls/issues/96
if net.ParseIP(config.ServerName) != nil {
err := uconn.RemoveSNIExtension()
if err != nil {
uconn.Close()
return nil, err
}
}
// We must call Handshake before returning, or else the UConn may not
// actually use the selected ClientHelloID. It depends on whether a Read
// or a Write happens first. If a Read happens first, the connection
// will use the normal crypto/tls fingerprint. If a Write happens first,
// it will use the selected fingerprint as expected.
// https://github.com/refraction-networking/utls/issues/75
err = uconn.Handshake()
if err != nil {
uconn.Close()
return nil, err
}
return uconn, nil
}
func fetch(url string) error {
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.DialTLSContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
return utlsDialContext(ctx, network, addr, nil, utlsClientHelloID)
}
transport.ForceAttemptHTTP2 = true
client := http.Client{
Transport: transport,
}
resp, err := client.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
fmt.Println(resp.Proto)
return nil
}
func main() {
flag.Parse()
if flag.NArg() != 1 {
fmt.Fprintf(os.Stderr, "usage: %s URL\n", os.Args[0])
os.Exit(1)
}
err := fetch(flag.Arg(0))
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
$ go run -- demo.go https://cloudflare.com/
Get "https://cloudflare.com/": net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x01\x00\x00\x04\x00\x01\x00\x00\x00\x05\x00\xff\xff\xff\x00\x00\x04\b\x00\x00\x00\x00\x00\u007f\xff\x00\x00\x00\x00\b\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01"
exit status 1
Anyone solved this issue ?
If you don't need HTTP/2, you can already do this [1]. Which in most cases, you don't. You just tell the server that you can only handle HTTP/1.1 [2]. Further, while its not a strictly required extension [3]:
A new extension type (
application_layer_protocol_negotiation(16)) is defined and MAY be included by the client in its "ClientHello" message.
Its currently used by every single parrot [4], so I think its a safe bet that the hello will already have it. If not, I would say it should be fine to just add it to whatever hello you are trying to send.
- https://github.com/89z/format/blob/v1.35.9/crypto/crypto.go#L98-L120
- https://github.com/89z/format/blob/v1.35.9/crypto/parse.go#L81-L85
- https://rfc-editor.org/rfc/rfc7301.html
- https://github.com/refraction-networking/utls/blob/master/u_parrots.go
you can implement your own http.Tranport.Define a http.RoundTrip helps. Following code is the self implemented http.RoundTrip:
package fetch
import (
"bufio"
"fmt"
"net"
"net/http"
"sync"
utls "github.com/refraction-networking/utls"
"golang.org/x/net/http2"
)
func NewBypassJA3Transport(helloID utls.ClientHelloID) *BypassJA3Transport {
return &BypassJA3Transport{clientHello: helloID}
}
type BypassJA3Transport struct {
tr1 http.Transport
tr2 http2.Transport
mu sync.RWMutex
clientHello utls.ClientHelloID
}
func (b *BypassJA3Transport) RoundTrip(req *http.Request) (*http.Response, error) {
switch req.URL.Scheme {
case "https":
return b.httpsRoundTrip(req)
case "http":
return b.tr1.RoundTrip(req)
default:
return nil, fmt.Errorf("unsupported scheme: %s", req.URL.Scheme)
}
}
func (b *BypassJA3Transport) httpsRoundTrip(req *http.Request) (*http.Response, error) {
port := req.URL.Port()
if port == "" {
port = "443"
}
conn, err := net.Dial("tcp", fmt.Sprintf("%s:%s", req.URL.Host, port))
if err != nil {
return nil, fmt.Errorf("tcp net dial fail: %w", err)
}
defer conn.Close() // nolint
tlsConn, err := b.tlsConnect(conn, req)
if err != nil {
return nil, fmt.Errorf("tls connect fail: %w", err)
}
httpVersion := tlsConn.ConnectionState().NegotiatedProtocol
switch httpVersion {
case "h2":
conn, err := b.tr2.NewClientConn(tlsConn)
if err != nil {
return nil, fmt.Errorf("create http2 client with connection fail: %w", err)
}
defer conn.Close() // nolint
return conn.RoundTrip(req)
case "http/1.1", "":
err := req.Write(tlsConn)
if err != nil {
return nil, fmt.Errorf("write http1 tls connection fail: %w", err)
}
return http.ReadResponse(bufio.NewReader(tlsConn), req)
default:
return nil, fmt.Errorf("unsuported http version: %s", httpVersion)
}
}
func (b *BypassJA3Transport) getTLSConfig(req *http.Request) *utls.Config {
return &utls.Config{
ServerName: req.URL.Host,
InsecureSkipVerify: true,
}
}
func (b *BypassJA3Transport) tlsConnect(conn net.Conn, req *http.Request) (*utls.UConn, error) {
b.mu.RLock()
tlsConn := utls.UClient(conn, b.getTLSConfig(req), b.clientHello)
b.mu.RUnlock()
if err := tlsConn.Handshake(); err != nil {
return nil, fmt.Errorf("tls handshake fail: %w", err)
}
return tlsConn, nil
}
func (b *BypassJA3Transport) SetClientHello(hello utls.ClientHelloID) {
b.mu.Lock()
defer b.mu.Unlock()
b.clientHello = hello
}
and set Tranport is enough:
var client = &http.Client{
Timeout: time.Second * 30,
Transport: NewBypassJA3Transport(utls.HelloChrome_102),
}
if it helps, I came up with the below implementation of http.RoundTripper. It only supports HTTP/1.1, but might be helpful to some users:
package tls
import (
"bufio"
"github.com/refraction-networking/utls"
"net"
"net/http"
)
type Transport struct {
Conn *tls.UConn
Spec tls.ClientHelloSpec
}
func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error) {
config := tls.Config{ServerName: req.URL.Host}
conn, err := net.Dial("tcp", req.URL.Host+":443")
if err != nil {
return nil, err
}
t.Conn = tls.UClient(conn, &config, tls.HelloCustom)
if err := t.Conn.ApplyPreset(&t.Spec); err != nil {
return nil, err
}
if err := req.Write(t.Conn); err != nil {
return nil, err
}
return http.ReadResponse(bufio.NewReader(t.Conn), req)
}
Don't know if anyone is still concerned with this stale issue, but if the purpose is simply for building HTTP Clients with uTLS as the tls stack, imroc/req seemed to be a good choice. See https://github.com/imroc/req/discussions/198 for multiple different ways you may use uTLS with imroc/req.
Closing this for now, but if issue not considered resolved we may reopen it.