Allow blacklisting HTTPS requests without connecting to server first

[jordanglassman]

This is a problem because these sorts of pathological situations are exactly why I started using browsermob-proxy. Workaround is to use AdvancedHostResolver#remapHost instead, but this is unsightly and might fail in some corner cases.

I think it probably has something to do with the order the HttpFiltersSourceAdapters are applied in BrowerMobProxyServer, but I'm new to this code base. I think either ResolvedHostnameCacheFilter or HttpsOriginalHostCaptureFilter requires a valid SSL socket to be created before the blacklist is applied in BlacklistFilter.

Using bmp 2.1.1 and selenium 2.53.1.

Test case:

    @Test
    public void bugReport() throws Exception {
        final BrowserMobProxyServer server = new BrowserMobProxyServer();
        server.blacklistRequests(".*google\\.com.*", 204);
        server.blacklistRequests(".*licdn\\.com.*", 204);
        server.start();

        final Proxy proxy = ClientUtil.createSeleniumProxy(server);
        final DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability(PROXY, proxy);
        final FirefoxDriver driver = new FirefoxDriver(capabilities);

        // works fine
        driver.get("https://www.google.com/");

        // java.net.SocketException
        driver.get("https://media-vnsh.licdn.com/");
    }

DEBUG logs:

2016-07-21 17:04:32,868 [:main] DEBUG ImpersonatingMitmManager - Allowed ciphers for proxy connections to upstream servers (some ciphers may not be available): [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2016-07-21 17:04:32,869 [:main] DEBUG ImpersonatingMitmManager - Allowed ciphers for client connections to proxy (some ciphers may not be available): [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_SHA, TLS_ECDH_ECDSA_WITH_RC4_128_SHA, TLS_ECDH_RSA_WITH_RC4_128_SHA, SSL_RSA_WITH_RC4_128_MD5, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2016-07-21 17:04:32,990 [:main] INFO  DefaultHttpProxyServer - Starting proxy at address: 0.0.0.0/0.0.0.0:0
2016-07-21 17:04:33,049 [:main] INFO  DefaultHttpProxyServer - Proxy listening with TCP transport
2016-07-21 17:04:33,454 [:main] INFO  DefaultHttpProxyServer - Proxy started at address: /0:0:0:0:0:0:0:0:37211
2016-07-21 17:04:36,264 [:LittleProxy-0-ClientToProxyWorker-1] DEBUG ImpersonatingMitmManager - Impersonated certificate for tiles-cloudfront.cdn.mozilla.net in 462ms
2016-07-21 17:04:36,349 [:LittleProxy-0-ClientToProxyWorker-0] DEBUG ImpersonatingMitmManager - Impersonated certificate for www.google.com in 547ms
2016-07-21 17:04:57,382 [:LittleProxy-0-ProxyToServerWorker-2] INFO  ProxyToServerConnection - (CONNECTING) [id: 0x6af86dfa]: Connection to upstream server failed
java.net.SocketException: Socket is not connected: no further information: media-vnsh.licdn.com/197.96.19.88:443
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:545)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
    at java.lang.Thread.run(Thread.java:745)
2016-07-21 17:04:57,444 [:LittleProxy-JVM-shutdown-hook] INFO  DefaultHttpProxyServer - Shutting down proxy server immediately (non-graceful)
2016-07-21 17:04:57,444 [:LittleProxy-JVM-shutdown-hook] INFO  DefaultHttpProxyServer - Closing all channels (non-graceful)
2016-07-21 17:04:57,450 [:LittleProxy-JVM-shutdown-hook] INFO  ServerGroup - Shutting down server group event loops (non-graceful)
2016-07-21 17:04:57,452 [:LittleProxy-JVM-shutdown-hook] INFO  DefaultHttpProxyServer - Done shutting down proxy server

[jekh]

Yes, you've found an unfortunate limitation of BMP that stems from LittleProxy's ConnectionFlow. There's a refactor of CF in the works that will solve this problem, but it won't be released until later this year. Actual HTTP requests (other than the CONNECT) are blacklisted, but the initial CONNECT still must be successful.

As you note, it's possible to work around it by using the AdvancedHostResolver to remap the host. That is less than an ideal solution, and will be addressed after the ConnectionFlow refactor.

[SupernaviX]

@jekh, is there any progress on that CF rewrite?

[jekh]

Not yet, unfortunately. I'm on the verge of releasing the next version of the REST API for comment. ConnectionFlow refactor is the next big thing on my list though.

[rafaelsilverioit]

Any updates on this issue? I'm having problems with a page that takes so long to load and I want to blacklist it, but the page is under HTTPS and is still being loaded.

And AdvancedHostResolver is not an option, since I still need to open the HTTPS website.

[andreyrusanov]

Any updates, guys? Having problems with blacklisting that doesn't work for https is really upset thing.. @jekh

[andreyrusanov]

@lightbody perhaps you can give any update on it? Is it possible to implement this feature?

[s-belichenko]

Hi all. Any updates?

[goagaalex]

Hello! Did anyone manage to resolve this problem independently from the browsermob-proxy ? Thank you!

[zhangjunqiang]

Hi all. Any updates?