htmlunit-driver
htmlunit-driver copied to clipboard
SeleniumHQ
Dependency Issue
I have a project using HTMLUnit Driver that is giving me dependency issues using 2.52. Below is my dependencies for selenium and htmlunit and the stacktrace.
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>3.141.59</version>
</dependency>
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoSuchFieldError: URL_ABOUT_BLANK
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1075)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at com.torchai.service.filter.ApiAuthorizationFilter.doFilterNexus(ApiAuthorizationFilter.java:61)
at com.torchai.service.filter.AuthFilter.doFilterInternal(AuthFilter.java:33)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchFieldError: URL_ABOUT_BLANK
at org.openqa.selenium.htmlunit.HtmlUnitDriver.
Looks like not matching jar files - htmlunit-driver 2.52.0 requires htmlunit 2.52.0 (There was some refactoring in the past regarding the URL_ABOUT_BLANK constant)
Can you please check if you have (only) the right versions in your classpath.
Having this same issue as well, in a really simple example in kotlin, but within a spring boot project. The TL;DR; is that it looks like an older dependency is getting swapped in somewhere from a spring dependency maybe?
fun scrapeBrowserTest() {
val driver: WebDriver = HtmlUnitDriver()
val wait = WebDriverWait(driver, Duration.ofSeconds(10).toSeconds())
try {
driver["https://amazon.com"]
val login = driver.findElement(By.name("nav-link-accountList"))
println(login.text)
} finally {
driver.quit()
}
}
gives me:
Exception in thread "main" java.lang.NoSuchFieldError: URL_ABOUT_BLANK
at org.openqa.selenium.htmlunit.HtmlUnitDriver.<init>(HtmlUnitDriver.java:277)
at org.openqa.selenium.htmlunit.HtmlUnitDriver.<init>(HtmlUnitDriver.java:227)
at org.openqa.selenium.htmlunit.HtmlUnitDriver.<init>(HtmlUnitDriver.java:198)
using only the following dependencies relevant to this execution:
implementation group: 'org.seleniumhq.selenium', name: 'htmlunit-driver', version: '2.52.0'
According to my dependency chain for 2.52.0, there are some substitutions coming from somewhere:
+--- org.seleniumhq.selenium:htmlunit-driver:2.52.0
| +--- org.seleniumhq.selenium:selenium-api:3.141.59
| +--- org.seleniumhq.selenium:selenium-support:3.141.59
| | +--- org.seleniumhq.selenium:selenium-api:3.141.59
| | +--- org.seleniumhq.selenium:selenium-remote-driver:3.141.59
| | | +--- org.seleniumhq.selenium:selenium-api:3.141.59
| | | +--- net.bytebuddy:byte-buddy:1.8.15 -> 1.10.14
| | | +--- org.apache.commons:commons-exec:1.3
| | | +--- com.google.guava:guava:25.0-jre (*)
| | | +--- com.squareup.okhttp3:okhttp:3.11.0 -> 3.14.9
| | | | \--- com.squareup.okio:okio:1.17.2
| | | \--- com.squareup.okio:okio:1.14.0 -> 1.17.2
| | +--- net.bytebuddy:byte-buddy:1.8.15 -> 1.10.14
| | +--- org.apache.commons:commons-exec:1.3
| | +--- com.google.guava:guava:25.0-jre (*)
| | +--- com.squareup.okhttp3:okhttp:3.11.0 -> 3.14.9 (*)
| | \--- com.squareup.okio:okio:1.14.0 -> 1.17.2
| \--- net.sourceforge.htmlunit:htmlunit:2.52.0 -> 2.40.0 !!!!!!!! think this is the issue..
| +--- xalan:xalan:2.7.2
| | \--- xalan:serializer:2.7.2
| +--- org.apache.commons:commons-lang3:3.10
| +--- org.apache.commons:commons-text:1.8
| +--- org.apache.httpcomponents:httpmime:4.5.12
| | \--- org.apache.httpcomponents:httpclient:4.5.12 (*)
| +--- net.sourceforge.htmlunit:htmlunit-core-js:2.40.0
| +--- net.sourceforge.htmlunit:neko-htmlunit:2.40.0
| | \--- xerces:xercesImpl:2.12.0
| | \--- xml-apis:xml-apis:1.4.01
| +--- net.sourceforge.htmlunit:htmlunit-cssparser:1.5.0
| +--- commons-io:commons-io:2.6
| +--- commons-logging:commons-logging:1.2
| +--- commons-net:commons-net:3.6
| +--- org.brotli:dec:0.1.2
| +--- com.shapesecurity:salvation:2.7.1
| \--- org.eclipse.jetty.websocket:websocket-client:9.4.28.v20200408 -> 9.4.31.v20200723
| +--- org.eclipse.jetty:jetty-client:9.4.31.v20200723
| | +--- org.eclipse.jetty:jetty-http:9.4.31.v20200723
| | | +--- org.eclipse.jetty:jetty-util:9.4.31.v20200723
| | | \--- org.eclipse.jetty:jetty-io:9.4.31.v20200723
| | | \--- org.eclipse.jetty:jetty-util:9.4.31.v20200723
| | \--- org.eclipse.jetty:jetty-io:9.4.31.v20200723 (*)
| +--- org.eclipse.jetty:jetty-xml:9.4.31.v20200723
| | \--- org.eclipse.jetty:jetty-util:9.4.31.v20200723
| +--- org.eclipse.jetty:jetty-util:9.4.31.v20200723
| +--- org.eclipse.jetty:jetty-io:9.4.31.v20200723 (*)
| \--- org.eclipse.jetty.websocket:websocket-common:9.4.31.v20200723
| +--- org.eclipse.jetty.websocket:websocket-api:9.4.31.v20200723
| +--- org.eclipse.jetty:jetty-util:9.4.31.v20200723
| \--- org.eclipse.jetty:jetty-io:9.4.31.v20200723 (*)
If, however, I use the dependencies together:
implementation group: 'net.sourceforge.htmlunit', name: 'htmlunit', version: '2.52.0'
implementation group: 'org.seleniumhq.selenium', name: 'htmlunit-driver', version: '2.52.0'
then the right dependency gets used and not swapped. Reverting to 2.50.0 and using only the htmlunit-driver dependency also worked for me, probably related to that URL_ABOUT_BLANK reference.
This is almost certainly a transitive dependency problem. For my own project, I exclude all of the drivers brought in by selenium-server to avoid the potential conflicts:
<dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-server</artifactId>
<exclusions>
<exclusion>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-chrome-driver</artifactId>
</exclusion>
<exclusion>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-edge-driver</artifactId>
</exclusion>
<exclusion>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-firefox-driver</artifactId>
</exclusion>
<exclusion>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-ie-driver</artifactId>
</exclusion>
<exclusion>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-opera-driver</artifactId>
</exclusion>
<exclusion>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
</exclusion>
<exclusion>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-safari-driver</artifactId>
</exclusion>
<exclusion>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>htmlunit-driver</artifactId>
</exclusion>
<exclusion>
<groupId>net.sourceforge.htmlunit</groupId>
<artifactId>htmlunit</artifactId>
</exclusion>
</exclusions>
</dependency>
Also, I use dependency management to control the versions of transitive dependencies. This is good practice that eliminates a lot of headaches.
To be abundantly clear, I am not sure this is an issue on the htmlunit-driver side. I agree it is a transitive dependency issue, and I think it is an artifact of using some spring library that happens to use an old ~2.40 version of htmlunit directly, which is getting swapped out underneath htmlunit-driver's otherwise bundled dependency.
There was really a change for version 2.51.0 that might be related to this.
https://github.com/SeleniumHQ/htmlunit-driver/commit/22d740365f3d60bf18bfb75c05aca154ed855cb5
But the current HtmlUnitDriver uses UrlUtils.URL_ABOUT_BLANK and this is available in HtmlUnit starting from version 2.47.0. The old WebClient.URL_ABOUT_BLANK is also still available for backward compatibility.
Still looks to me like you have an old version of HtmlUnit somewhere in your classpath.
Unfortunately no. For the purpose of my project I just reverted back to version 2.50.
net.sourceforge.htmlunit:htmlunit:2.52.0 -> 2.40.0 !!!!!!!! think this is the issue..
I think you are right, this might be the problem. Looks like some other lib depends on this old version. Maybe you can exclude HtmlUnit from the other.