browsermob-proxy
browsermob-proxy copied to clipboard
Embedded mode with selenium doesn't load website
When using embedded mode in a selenium test, websites will timeout and not load. I have tried on google, npr.org, yahoo.com, etc. I've tried a couple things to see if it makes a difference but no luck so far: setting setUseECC to true, setTrustAllServers to true. I've tried on 2 different machines and same results. Is there something I'm missing?
Java v8 selenium v3.3.1 bmp v2.1.4
Here is my test case:
// Create bmp proxy
BrowserMobProxyServer proxy = new BrowserMobProxyServer();
// proxy.start(0);
try {
proxy.start(9091, InetAddress.getLocalHost());
} catch (UnknownHostException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
proxy.setHarCaptureTypes(CaptureType.getAllContentCaptureTypes());
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
proxy.setUseEcc(true);
// Pass the proxy into desired capabilities
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
DesiredCapabilities caps = new DesiredCapabilities();
caps = DesiredCapabilities.chrome();
caps.setCapability(CapabilityType.PROXY, seleniumProxy);
// Turning off pop-up blocking
ChromeOptions options = new ChromeOptions();
options.addArguments("test-type");
options.addArguments("disable-popup-blocking");
caps.setCapability(ChromeOptions.CAPABILITY, options);
// Create the browser session
File file = new File(this.getClass().getResource(Constants.DRIVERS_PATH_LOCAL + "ChromeDriver.exe").getPath());
System.setProperty("webdriver.chrome.driver", file.getAbsolutePath());
WebDriver driver = new ChromeDriver(caps);
// Create the har file
proxy.newHar("SalesforceAddCandidate");
// load website stuff
driver.get("http://www.npr.org/");
Har har = proxy.getHar();
File harFile = new File("C:\\temp\\SalesforceAddCandidate");
try {
har.writeTo(harFile);
} catch (IOException ex) {
System.out.println(ex.toString());
}
proxy.stop();
Facing same issue. tried different option looks like proxy is not working properly
See if this helps: https://techblog.dotdash.com/selenium-browsermob-integration-c35f4713fb59
if you are running your test locally to BMP server you may just want us proxy.start(9091) instead of "proxy.start(9091, InetAddress.getLocalHost());" Try and post the results
I'm facing the same problem @jessicamarshall @Ritesh-Yadav . I already now what my and probably your problem is. When you load the latest version of BrowserMob-Proxy via a Maven repository some classes of it use the method getHostText() from Google Guava. However, this method has been renamed to getHost() since version 21 (I think). In addition when you load Selenium from Maven it loads the latest version of Guava (currently 23), where the name of the method changed to getHost() and so you become a NoSuchMethodError.
The weird thing is that this Git-repository is already updated to Guava 22 and it uses the new-named getHost() method, but when you load the latest version from mavenCentral, you get an old version with getHostText().
I also tried to directly download this git repository via https://jitpack.io/, but I got the same error like above.
A little tip: When calling BasicConfigurator.configure();
before creating the proxy, logging to console is enabled and so you will see the NoSuchMethodError
@nhoellinger - I fixed that problem yesterday by excluding new guava version from my build and included guava 21.0. Now I can capture all the requests.
Thanks. Ritesh
I've been stuck on this issue for a few days now. I'm hoping somebody can help resolve it.
`public class bmp_Selenium {
@Test
public void test() {
String chromedriverPath = System.getProperty("user.dir") + "/chromedriver";
System.setProperty("webdriver.chrome.driver", chromedriverPath);
BrowserMobProxy proxy = getProxyServer(); //getting browsermob proxy
System.out.println("BrowserMob Proxy running on port: " + proxy.getPort());
Proxy seleniumProxy = getSeleniumProxy(proxy);
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);
WebDriver driver = new ChromeDriver(capabilities);
proxy.newHar("Har File");
try {
driver.get("https://www.google.com");
Har har = proxy.getHar();
// Write HAR Data in a File
String harFilePath = System.getProperty("user.dir") + "/hars/ww.har";
File harFile = new File(harFilePath);
try {
har.writeTo(harFile);
} catch (IOException ex) {
System.out.println (ex.toString());
System.out.println("Could not find file " + harFilePath);
}
List<HarEntry> entries = har.getLog().getEntries();
for (HarEntry entry : entries) {
System.out.println("Request URL: " + entry.getRequest().getUrl());
System.out.println("Entry response status: " + entry.getResponse().getStatus());
System.out.println("Entry response text: " + entry.getResponse().getStatusText());
}
}
catch(Exception e) {
System.out.println("Exception caught. Stopping proxy and webdriver.\nException: " + e);
}
finally
{
proxy.stop();
driver.quit();
}
}
public Proxy getSeleniumProxy(BrowserMobProxy proxyServer) {
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxyServer);
try {
String hostIp = Inet4Address.getLocalHost().getHostAddress();
seleniumProxy.setHttpProxy(hostIp + ":" + proxyServer.getPort());
seleniumProxy.setSslProxy(hostIp + ":" + proxyServer.getPort());
} catch (Exception e) {
e.printStackTrace();
Assert.fail("invalid Host Address");
}
return seleniumProxy;
}
public BrowserMobProxy getProxyServer() {
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.setHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
proxy.start();
return proxy;
}
}`
No exceptions are being thrown. A Chrome browser is launched, and it displays the following when trying to navigate to the Google URL
"This page isn’t working www.google.com didn’t send any data. ERR_EMPTY_RESPONSE"
Console output is:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. BrowserMob Proxy running on port: 9091 Starting ChromeDriver 2.33.506106 (8a06c39c4582fbfbab6966dbb1c38a9173bfb1a2) on port 35307 Only local connections are allowed. Dec 01, 2017 2:27:47 PM org.openqa.selenium.remote.ProtocolHandshake createSession INFO: Detected dialect: OSS
Curious , is this a typo or you missed typing .exe
String chromedriverPath = System.getProperty("user.dir") + "/chromedriver";
should be
String chromedriverPath = System.getProperty("user.dir") + "/chromedriver.exe";
Other than this dont see any thing wrong with the code and it works for me.
what are the versions for Browsermob, selenium, chrome driver, chrome browser you are using?
Suggest to make few changes as you are using deprecated way to create chrome driver and seem to do few redundant proxy settings.
Suggestion 1:
like you dont need this as seleniumProxy object can be passed into a capability directly
try {
String hostIp = Inet4Address.getLocalHost().getHostAddress();
seleniumProxy.setHttpProxy(hostIp + ":" + proxyServer.getPort());
seleniumProxy.setSslProxy(hostIp + ":" + proxyServer.getPort());
} catch (Exception e) {
e.printStackTrace();
Assert.fail("invalid Host Address");
}
Suggestion 2: create chrome driver like this using ChromeOptions. setting Capabilities in the ChromeDriver constructor is deprecated
ChromeOptions options = new ChromeOptions();
options.setCapability(CapabilityType.PROXY, seleniumProxy);
WebDriver driver = new ChromeDriver(capabilities);
@mediga Thanks for the quick reply.
Chromedriver is 2.33. Chrome version 62. I'm on MacOS Sierra 10.12.6
Here's my full maven dependencies which list the versions I'm using:
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.4</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.4</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.7.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>23.0</version>
</dependency>
I believe I implemented that suggestion and your 1st one, but it didn't change anything. Below is my updated code.
` public class bmp_Selenium {
@Test
public void test() throws UnknownHostException {
String chromedriverPath = System.getProperty("user.dir") + "/chromedriver";
System.setProperty("webdriver.chrome.driver", chromedriverPath);
BrowserMobProxy proxy = getProxyServer(); //getting browsermob proxy
System.out.println("BrowserMob Proxy running on port: " + proxy.getPort());
Proxy seleniumProxy = getSeleniumProxy(proxy);
ChromeOptions options = new ChromeOptions();
options.setCapability(CapabilityType.PROXY, seleniumProxy);
WebDriver driver = new ChromeDriver(options);
proxy.newHar("Har File");
try {
driver.get("https://www.google.com");
Har har = proxy.getHar();
// Write HAR Data in a File
String harFilePath = System.getProperty("user.dir") + "/hars/ww.har";
File harFile = new File(harFilePath);
try {
har.writeTo(harFile);
} catch (IOException ex) {
System.out.println (ex.toString());
System.out.println("Could not find file " + harFilePath);
}
List<HarEntry> entries = har.getLog().getEntries();
for (HarEntry entry : entries) {
System.out.println("Request URL: " + entry.getRequest().getUrl());
System.out.println("Entry response status: " + entry.getResponse().getStatus());
System.out.println("Entry response text: " + entry.getResponse().getStatusText());
}
}
catch(Exception e) {
System.out.println("Exception caught. Stopping proxy and webdriver.\nException: " + e);
}
finally
{
proxy.stop();
driver.quit();
}
}
public Proxy getSeleniumProxy(BrowserMobProxy proxyServer) {
Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxyServer);
return seleniumProxy;
}
public BrowserMobProxy getProxyServer() throws UnknownHostException {
BrowserMobProxy proxy = new BrowserMobProxyServer();
proxy.setTrustAllServers(true);
proxy.setHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);
proxy.start();
return proxy;
}
}`
I'm really stuck here, any other suggestions you may have are appreciated
try this. hope that works
replace this
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.4</version>
<scope>test</scope>
</dependency>
with
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
</dependency>
@mediga Problem solved! Thank you. When it does load the Google webpage now though, the browser says "Not Secure". Is that expected?
And also, if you wouldn't mind going into a little more detail, just so I can understand the root cause, that'd be appreciated too. Thanks again.
-
Root cause seems to be either the
<scope>test</scope>
of the dependency which is most likely the cause i think or the version of Browsermob -
To avoid 'Not Secure"
try these two suggestions
a) add options.setAcceptInsecureCerts(true);
to your code.
b)download ca-certificate-rsa.cer from
https://github.com/lightbody/browsermob-proxy/blob/master/browsermob-core/src/main/resources/sslSupport/ca-certificate-rsa.cer
c) import ca-certificate-rsa.cer into Chrome (Settings -> Manage Certificates -> go to Trusted Root Certification Authorities -> import by pointing to ca-certificate-rsa.cer file.
this should do it
I've been having the same issue, duplicated @cafshari's code and matched as close as possible to his setup, but I can't for the life of me get the damn thing to work.
No matter what combination of selenium, BMP, chromedriver, guava, etc versions - I can't load any page. Been beating my head against the wall for the last 6 hours with absolutely no signs of life or luck :(
I guess the only difference is I'm including the dependencies into the jar and not running as a test?
<artifactSet>
<includes>
<include>net.lightbody.bmp:browsermob-core</include>
<include>net.lightbody.bmp:littleproxy</include>
<include>net.lightbody.bmp:mitm</include>
<include>org.slf4j:slf4j-api</include>
<include>org.bouncycastle:bcprov-jdk15on</include>
<include>org.bouncycastle:bcpkix-jdk15on</include>
<include>dnsjava:dnsjava</include>
<include>io.netty:netty-all</include>
<include>com.fasterxml.jackson.core:jackson-core</include>
<include>com.fasterxml.jackson.core:jackson-databind</include>
<include>com.fasterxml.jackson.core:jackson-annotations</include>
<include>org.seleniumhq.selenium:selenium-chrome-driver</include>
<include>org.seleniumhq.selenium:selenium-remote-driver</include>
<include>org.seleniumhq.selenium:selenium-api</include>
<include>org.seleniumhq.selenium:selenium-java</include>
<include>org.apache.httpcomponents:httpclient</include>
<include>org.apache.httpcomponents:httpcore</include>
<include>org.apache.commons:commons-exec</include>
<include>org.apache.commons:commons-lang</include>
<include>org.apache.logging.log4j:log4j-api</include>
<include>org.apache.logging.log4j:log4j-core</include>
<include>com.google.code.gson:gson</include>
<include>com.google.guava:guava</include>
<include>net.java.dev.jna:jna-platform</include>
<include>commons-logging</include>
</includes>
</artifactSet>
<dependencies>
<dependency>
<groupId>net.lightbody.bmp</groupId>
<artifactId>browsermob-core</artifactId>
<version>2.1.5</version>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.3.4</version>
</dependency>
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.8.1</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>23.0</version>
</dependency>
</dependencies>
No exceptions, BMP appears to startup fine.
Yeah I still can't get it to work at all :/ Willing to toss a few bucks to someone who has any idea how to get this working
can you upload your project, so that i can try running it on my side. It will be good if you can trim it to bare minimum to reproduce the issue.
Sure, the code here is basically exactly like @cafshari's. Trimmed it to be pretty minimal, make sure the "drivers" folder is in the target folder. Used IntelliJ for this, so you should be able to open the project and it'll use maven. I didn't add any repos to my pom, so hopefully your local repo already has the versions it needs.
Thanks for providing the sample project.
I was able to run your project AS-IS and it worked fine. I am using the following software versions. what are your software versions?
Chrome: Version 63.0.3239.84 (Official Build) (64-bit) Chrome driver: 2.34 (2.34.522940 ) OS : Windows 10
also try to enable chrome logging by doing this and attach the log here.
System.setProperty("webdriver.chrome.logfile","c:\\mystuff\\chromedriver_bmptest.log");
System.setProperty("webdriver.chrome.verboseLogging", "true");
If we cannot find anything in the chrome log, try enabling logging on the driver like this
LoggingPreferences logs = new LoggingPreferences();
logs.enable(LogType.BROWSER, Level.ALL);
logs.enable(LogType.CLIENT, Level.ALL);
logs.enable(LogType.DRIVER, Level.ALL);
logs.enable(LogType.PERFORMANCE, Level.ALL);
logs.enable(LogType.PROFILER, Level.ALL);
logs.enable(LogType.SERVER, Level.ALL);
options.setCapability(CapabilityType.LOGGING_PREFS, logs);
//write these logs to file
Logs logs = driver.manage().logs();
LogEntries driverLogEntries = logs.get(LogType.DRIVER);
LogEntries browserLogEntries = logs.get(LogType.BROWSER );
LogEntries clientLogEntries = logs.get(LogType.CLIENT);
LogEntries performanceLogEntries = logs.get(LogType.PERFORMANCE);
Thanks for testing it out!
Here are my versions: Chrome: Version 63.0.3239.108 (Official Build) (64-bit) Chrome Driver: 2.34 OS: Windows 10
And here's the logs. I can see the proxy is set, the port matches the BMP one. Hostname is PC hostname, which is pingable (resolves to local IPV6 address however)
can you change the url in the code from
http://www.google.com
to
https://www.google.com
if still does not work, try with
http://www.cnn.com
let me know how it goes please and send log if the problem persists with these two tests.
Same thing for both (ERR_EMPTY_RESPONSE)
can you attach the log file?
does your computer have any other network proxy or a vpn or a firewall which could be preventing browsermob proxy from reaching internet?
From the logs, it does look like traffic reaches browsermob and does not go further from there.
[3.675][DEBUG]: DEVTOOLS EVENT Log.entryAdded {
"entry": {
"level": "error",
"networkRequestId": "50380.10",
**"source": "network",
"text": "Failed to load resource: net::ERR_EMPTY_RESPONSE",**
"timestamp": 1514750262360.31,
"url": "http://www.google.com/"
}
"text": "Failed to load resource: net::ERR_EMPTY_RESPONSE",
"id": "(12480AB0EA7B983AC062823B8FB1A356)",
"loaderId": "50380.2",
"mimeType": "text/html",
**"securityOrigin": "://",
**"unreachableUrl": "http://www.google.com/",**
"url": "chrome-error://chromewebdata/"**
Nope, no proxies.
I disabled Windows firewall and disabled all firewall policies on my router.
Ran as admin, tried another Win 10 PC.. No luck. Tried another ISP entirely, no luck. Tried another ISP on another laptop, nothing.
This program hates me, I'm cursed lol.
would you be able to try these tests:
- install charles proxy with selenium sending traffic through charles proxy. if this works try #2. if this does not work, then for sure we will know the issue is not specific to browsermob.
- start standalone browsermob proxy with selenium sending traffic through standalone instance . if does not work, try to turn on trace logging in <browermob_installation>/bin/conf/bmp-logging.yaml file
- if standalone works, the next logical step would be to turn on slf4j logging for embedded browsermob and further troubleshoot.
sorry, since i cannot reproduce at my end, I am not able to provide much help but ask for more information
To enable logging for Embedded mode:
Maven dependency:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.8.0-beta0</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.8.0-beta0</version>
</dependency>
my log4j.properties
# Root logger option
log4j.rootLogger=TRACE, file,stdout
log_root=C:\\Tools\\Application\\log4j
log4j.logger.net.lightbody.bmp=TRACE
# Direct log messages to a log file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=${log_root}/my.log
log4j.appender.file.MaxFileSize=5MB
log4j.appender.file.MaxBackupIndex=20
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
No worries, thanks for the help!
-
Am able to get traffic flowing through Charles Proxy with: options.addArguments("--proxy-server=localhost:8888"); yet this doesn't work for BMP: options.addArguments("--proxy-server=localhost:" + proxy.getPort());
-
Can't connect to standalone BMP with: options.addArguments("--proxy-server=localhost:9090"); (or 127.0.0.1 / 0.0.0.0 / diff ports / diff sites etc)
BMP Standlone is running:
[INFO 2018-01-01T23:35:49,378 net.lightbody.bmp.proxy.Main] (main) Starting BrowserMob Proxy version 2.1.4
[INFO 2018-01-01T23:35:49,466 org.eclipse.jetty.util.log] (main) jetty-7.x.y-SNAPSHOT
[INFO 2018-01-01T23:35:49,745 org.eclipse.jetty.util.log] (main) started o.e.j.s.ServletContextHandler{/,null}
[INFO 2018-01-01T23:35:50,204 org.eclipse.jetty.util.log] (main) Started [email protected]:9090
Yet:
Interesting output from trace log level:
[DEBUG 2018-01-01T23:41:34,084 org.eclipse.jetty.util.log] (qtp530696881-20 - www.google.com:443) REQUEST www.google.com:443 on org.eclipse.jetty.server.nio.SelectChannelConnector$2@63cf56ba
[DEBUG 2018-01-01T23:41:34,085 org.eclipse.jetty.util.log] (qtp530696881-20 - www.google.com:443) sessionManager=org.eclipse.jetty.server.session.HashSessionManager@ee86bcb
[DEBUG 2018-01-01T23:41:34,085 org.eclipse.jetty.util.log] (qtp530696881-20 - www.google.com:443) session=null
[DEBUG 2018-01-01T23:41:34,088 org.eclipse.jetty.util.log] (qtp530696881-20 - www.google.com:443) servlet holder=
[DEBUG 2018-01-01T23:41:34,091 org.eclipse.jetty.util.log] (qtp530696881-20 - www.google.com:443) RESPONSE www.google.com:443 200
[DEBUG 2018-01-01T23:41:34,093 org.eclipse.jetty.util.log] (qtp530696881-21) EOF org.eclipse.jetty.io.EofException: null
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:321) ~[browsermob-dist-2.1.4.jar:?]
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) ~[browsermob-dist-2.1.4.jar:?]
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) ~[browsermob-dist-2.1.4.jar:?]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:520) [browsermob-dist-2.1.4.jar:?]
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) [browsermob-dist-2.1.4.jar:?]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:528) [browsermob-dist-2.1.4.jar:?]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_151]
200 OK response from google, yet EOF from Jetty/HttpParser?
- Standalone doesn't quite work, it kinda connects, but not really. So I'll leave it here for now.
Interesting that you are seeing org.eclipse.jetty package in the log and browsermob-dist-2.1.4.jar instead of 2.1.5. please attach the trace log file and your log4j.properties.
I made your sample project to use browsermob proxy 2.1.4 instead of 2.1.5 and can reproduce ERR_EMPTY_RESPONSE. If you are not using 2.1.5, you should switch.
Problem found!
While getting slf4j to work, I noticed it was spamming this in the trace log:
java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils
Turns out, I was shading: org.apache.commons:commons-lang
when I should have shaded org.apache.commons:commons-lang3
I'm not sure how or why it worked for you, but after adding the 3, it's working perfectly now! Remove the 3 and I get the same issue.
Thanks for helping, I would've figured that an exception like that would be visible normally.. but apparently not?
great news!! you definitely are not cursed :)
I am also facing this issue. Could someone please update on this?
Hello, I am also facing this problem, which has been solved in the following way.
After the request, the program pauses for 2 seconds.
code: Thread.sleep (2000);
When the interface is loaded, you can get the data.