How to ignore/bypass android apps's certificate pinning

[ghostlexly]

Hello,

The android apps have a certificate pinning that block mockttp. Here is the debug output:

TLS client error: {"failureCause":"cert-rejected","hostname":"api.twitter.com","remoteIpAddress":"::ffff:10.3.141.91","remotePort":45686,"tags":[],"timingEvents":{"startTime":1641902289465,"connectTimestamp":12868.375694,"failureTimestamp":12923.67577}}

I know that there is no way to bypass without jailbreaking the phone. So, i would love to find a way to ignore these requests from the proxy. If we can detect the header of the request coming from the apps, we can ignore the request then and make the app work without the proxy.

There is the code i have written so far:

http-combo-server.js

server.addListener('connect', function (req, resOrSocket) {
            if (resOrSocket instanceof net.Socket) {

               // if it's not a webbrowser, we can't proxy the request, ignore it
                if (!req.headers['user-agent'] || !req.headers['user-agent'].match("Mozilla")) {
                    var host = req.url.split(":")[0];
                    var port = req.url.split(":")[1];
    
                    var conn = net.connect({
                        port: port,
                        host: host,
                        allowHalfOpen: true
                    }, function(){
                        conn.on('finish', () => {
                          socket.destroy();
                        });
                        socket.on('close', () => {
                          conn.end();
                        });
                        socket.write('HTTP/1.1 200 OK\r\n\r\n', 'UTF-8', function(){
                          conn.pipe(socket);
                          socket.pipe(conn);
                        });
                    });
                    return false;
                }

                handleH1Connect(req, resOrSocket);
            }
            else {
                handleH2Connect(req, resOrSocket);
            }
        });

But it's not working, the TLS error happens before the connect.

If someone can help me on this i would be really gratefull

[pimterry]

This is harder said than done: there is no way to see any HTTP request data before the TLS connection is created, and you can't create the TLS connection if the client doesn't trust your certificate. This isn't a Mockttp or Node.js limitation - it's an unavoidable limitation of HTTPS. In all cases, first the TLS connection is created and the certificates checked, and only then later is any HTTP data sent on that connection, if it's trusted.

It is however theoretically possible to see the contents of the TLS client hello in plain text though, which includes the name of the server that the client wants to connect to (via SNI), in most cases (very very old clients don't include this, and also in the long-term future if ECH seriously takes off then that probably won't be possible for all connections either). You might want to look at this byte-by-byte TLS handshake explanation if that doesn't make sense to you.

Unfortunately though, Node.js's built-in TLS server doesn't let us do anything with this. All the standard TLS server does is take raw TCP connections and make TLS handshakes to create TLS connections for every one. You can't take an incoming raw connection, inspect the server name, and then proxy some connections untouched whilst creating intercepted TLS connections for others.

The only way to do this would be to extend httpolyglot to look at incoming TLS packets manually, parse the TLS client hello of those to extract the server name, and then handle connections differently with custom logic depending on the name you find.

Httpolyglot already does something similar in a very simple way (it looks at the first byte, and then opens either an HTTP, HTTP/2 or TLS connection depending on the traffic it sees) so you'd need to extend that to read the whole client hello with TLS is detected and parse that, find the server name if it's there, and then either continue with TLS setup (if the name doesn't match) or trigger a new callback for untouched proxying (if the name does match a domain you don't want to intercept). There's more details on how the current setup works here: https://httptoolkit.tech/blog/http-https-same-port/

Does that make sense? This would be a useful feature, so I'd be very happy to accept PRs for that if that's interesting to you! Note that all contributors to any of HTTP Toolkit's open source projects get free HTTP Toolkit Pro.

Alternatively, if an on-device fix to stop Twitter breaking while you intercept other apps would work for you, then you might want to look at HTTP Toolkit's own Android app. This acts as a fake VPN that redirects packets to send them to the proxy, but notably by using the VPN APIs it can intercept only specific applications, leaving others using the device's normal networking. The standard build of that app needs HTTP Toolkit to provide its config, but you could modify and rebuild it to add your settings there directly. There's more details about how this works here: https://httptoolkit.tech/blog/inspecting-android-http/.

[ghostlexly]

Thanks for the tips and detailled infos. I would be happy to help this repo with a solution on this problem.

There is a update on my work: I was able to make this work with Android + iPhone, even on Twitter app with node-http-mitm-proxy with this code:

proxy.onConnect(function(req,socket,head,callback) {
  var host = req.url.split(":")[0];
  var port = req.url.split(":")[1];

  console.log(req.headers['user-agent']);
  if (req.headers['user-agent'] && req.headers['user-agent'].match("Mozilla")) {
    console.log('Tunnel to', req.url);
    return callback();
  } else {
    // bypass
    var conn = net.connect({
      port: port,
      host: host,
      allowHalfOpen: true
    }, function(){
      conn.on('finish', () => {
        socket.destroy();
      });
      socket.on('close', () => {
        conn.end();
      });
      socket.write('HTTP/1.1 200 OK\r\n\r\n', 'UTF-8', function(){
        conn.pipe(socket);
        socket.pipe(conn);
      });
    });
  
    conn.on('error', function(err) {
      filterSocketConnReset(err, 'PROXY_TO_SERVER_SOCKET');
    });
    socket.on('error', function(err) {
      filterSocketConnReset(err, 'CLIENT_TO_PROXY_SOCKET');
    });
  }
});

I'm trying now to do the same on mockttp but i'm unable to find a way to make the httpolyglot keep the connection alive for the check to proceed. httpolyglot just close the connection with cert-rejected message.

[pimterry]

If that code works for you, then your client isn't making a direct HTTPS connection, it's making a plaintext or unpinned HTTP CONNECT request to the proxy, and then a subsequent HTTPS tunnel where the pinning is used.

In that case, you can see the initial CONNECT request data, you just can't see inside the tunnel, and that CONNECT data may indeed have useful info. This is a good point that I'd totally forgotten about!

This won't work for transparent proxying, where the client is unaware they're talking to a proxy. It sounds like that's OK for your use case though, and to be fair it probably works for most other cases too, and it's much easier to handle than parsing TLS by hand ourselves.

To support this, you want to look here: https://github.com/httptoolkit/mockttp/blob/c4a18bf15e2b6ed34397e0e243cdf1f2f69ccd94/src/server/http-combo-server.ts#L229-L287

That code handles CONNECT requests (for both HTTP/1 & HTTP/2) and intercepts the tunnelled connections. In both cases, the full flow that happens is:

1. A raw connection arrives, and the raw connection data goes to httpolyglot
2. Httpolyglot sniffs the first bytes of the connection

• If detects TLS, it sets up a TLS connection with a fake certificate for the requested server, and repeats this step on the data inside that connection
• Once it detects HTTP/1 or HTTP/2, it passes the connection back to the appropriate HTTP server

3. When the HTTP server sees that it's receiving a CONNECT request, it calls that connect event callback instead of the normal request listener
4. That callback reads the target URL, and replies with a successful response (telling the client that it has created the tunnel, and all new data will be sent direct to the target - this is a lie)
5. Then Mockttp just passes the connection back to httpolyglot, to start the whole process from step 1 using the data inside the tunnelled connection.

In your case, you want to add some logic before step 5: if the target URL matches some preconfigured list of hosts, then the connection really should be tunnelled to the target and not sent back to httpolyglot (i.e. you shouldn't call server.emit('connection', ...)).

Does that make sense?

[pimterry]

This is now supported in v3.5.0, using the tlsPassthrough option, like so:

mockServer = getLocal({
    https: {
        /* your HTTPS config */
        tlsPassthrough: [{ hostname: 'example.com' }]
    }
});

All requests to hostnames in that list will be spotted before Mockttp's TLS handshake begins, and forwarded upstream raw, bypassing TLS interception. You won't be able to see or match the request content at all, because it's not being intercepted, but you can hear about the existence & details of the tunnel itself by subscribing to the tls-passthrough-opened and tls-passthrough-closed events.

Thanks for filing this @ghostlexly!