MITM Exception: Upstream host null while intercepting Android Native app traffic using BrowserMob

[SagarBShinde]

Hi all, I have been working on a requirement to monitor and intercept native app traffic on mobile device with MITM proxy. I have decided to go with BrowserMob Proxy due to it's API support. Following is my set up BMP version 2.1.5 https://github.com/lightbody/browsermob-proxy Mobile device - OnePlus 2 Android v 6.0.1 Proxy set up on Mobile using Drony https://play.google.com/store/apps/details?id=org.sandroproxy.drony&hl=en_IN I have installed BMP SSL certificate on Mobile device Both Mobile and BMP host are on the same WiFi network I am not able to establish a connection between BMP and Mobile device. I have been able to monitor traffic via Charles Proxy however encoutering issues with the BMP set up Getting following error in BMP logs WARNING: An exception was thrown by org.littleshoot.proxy.impl.ConnectionFlow$2.operationComplete() net.lightbody.bmp.mitm.exception.MitmException: Error creating SSLEngine for connection to client to impersonate upstream host: null at net.lightbody.bmp.mitm.manager.ImpersonatingMitmManager.clientSslEngineFor(ImpersonatingMitmManager.java:227) at org.littleshoot.proxy.impl.ProxyToServerConnection$3.execute(ProxyToServerConnection.java:724) at org.littleshoot.proxy.impl.ConnectionFlow.doProcessCurrentStep(ConnectionFlow.java:140) at org.littleshoot.proxy.impl.ConnectionFlow.processCurrentStep(ConnectionFlow.java:128) at org.littleshoot.proxy.impl.ConnectionFlow.advance(ConnectionFlow.java:90) at org.littleshoot.proxy.impl.ConnectionFlowStep.onSuccess(ConnectionFlowStep.java:83) at org.littleshoot.proxy.impl.ConnectionFlow$2.operationComplete(ConnectionFlow.java:149) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:162) at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) at org.littleshoot.proxy.impl.ConnectionFlow.doProcessCurrentStep(ConnectionFlow.java:140) at org.littleshoot.proxy.impl.ConnectionFlow.access$000(ConnectionFlow.java:14) at org.littleshoot.proxy.impl.ConnectionFlow$1.run(ConnectionFlow.java:124) at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:73) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.NullPointerException at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:888) at com.google.common.cache.LocalCache.get(LocalCache.java:3849) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4712) at net.lightbody.bmp.mitm.manager.ImpersonatingMitmManager.getHostnameImpersonatingSslContext(ImpersonatingMitmManager.java:242) at net.lightbody.bmp.mitm.manager.ImpersonatingMitmManager.clientSslEngineFor(ImpersonatingMitmManager.java:223) ... 22 more Can anybody give pointers on how we can resolve this error?

[SagarBShinde]

@jekh - Can I request you to take a look at this ? Thanks in advance!

[eyal919]

any updates on this one? Thanks

[selenUser]

I successes connect BMP to mobile device and got all network console, but for all request i'm getting empty request and response with error "Unable to connect to host". I of course installed certificate on device, what I missed ?

[suitux]

Any updates on this?

[puigru]

I was also running into this issue from using browsermob with the Python requests package.

This is caused by clients that don't include a Host header for HTTP CONNECT requests. The HTTP/1.1 specification mandates that all requests should include a Host header. However, many clients are not complying in this regard. This might be because an older version, HTTP/1.0, does allow requests without that header. Hence servers that aim to support HTTP/1.0 should allow requests that do not include it.

This would need to be handled in HttpUtil#getHostFromRequest(). I made a quick patch on my end that simply prepends https to the URI for CONNECT requests on port 443. Not sure if that's the best way to handle it. I would open a PR but after fixing this I stumbled across another one (#835), which I cannot be bothered to dig into. Shame that this project is abandoned.

[petermertos]

@XXLuigiMario - can you please share your fix?

[petermertos]

Hope it can help someone. Inserted to HttpUtil#getHostFromRequest(): import io.netty.handler.codec.http.HttpMethod;

if (host == null && httpRequest.getMethod() == HttpMethod.CONNECT) { host = httpRequest.getUri().split(":")[0]; }