SeleniumHQ
Forms are submitted twice if Javascript is enabled
In com.gargoylesoftware.htmlunit.html.HtmlForm:submit the first HttpPost is done through Javascript call. Then webClient.download is called and a second HttpPost is done.
This behavior is causing a problem with Okta authentication server which redirect to password page for the first HttpPost but for the second one we get a redirection to the username prompt, causing a loop.
I tried a browser without JS_FORM_SUBMIT_FORCES_DOWNLOAD (Firefox) but because of some tests a second call is still made.
I think you should not do a second call when Javascript is enabled, no matter if JS_FORM_SUBMIT_FORCES_DOWNLOAD is supported or not. Or at least put an option to avoid a second download.
I have a workaround to make it work, I disable Javascript for this page, and reenable it after. Hopefully the login page don't need Javascript enabled, so it works.
@sesa501225 OMG can you please provide a sample html form to illustrate your case
Hello, Here is the Okta form
state: hKFo2SBQVDB3YlVQR2tXV3pBSTZsVDhzeVM4YlpLMEsxcl9ITKFur3VuaXZlcnNhbC1sb2dpbqN0aWTZIHIyVGpYRHZzTWU1MUZKQVVUQVBESVdta0RSNjJpcGxJo2NpZNkga3NOV29pVDBmZ1lodGdYTDFZMm5sNDFPTW0xR3B0SXo
username: xxxx
js-available: true
webauthn-available: true
is-brave: false
webauthn-platform-available: true
You can see that there is a state and Okta does not accept two requests with the same state. I guess it generates a state before login page, the save it in a database, and once it get a response remove it from database and send a redirect to password page, the second response it gets the state is no more in database and hence send a redirect back to login page with a new generated state.
@sesa501225 what i was asking for was more the client side of the story - i guess there is a html form in your browser. Maybe having a submit handler or something like that. Understanding the client side helps me to create a test case for that .
JS_FORM_SUBMIT_FORCES_DOWNLOAD
@sesa501225 Looks like your are still at version 3.x the JS_FORM_SUBMIT_FORCES_DOWNLOAD flag is no longer available since version 4.0. Any chance to check with the latest version?
I tested and did the workaround with 4.13.0, I get back to 3.64.0 because we are still using Java 8. The client side does not matter, you need a server which does not give the same response when called twice.
Here is the client form, even if I doubt it will have any utility
<form method="POST" class="cb97005fe _form-login-id" data-form-primary="true" data-disable-html-validations="true" novalidate="">
<button type="submit" name="action" class="ulp-hidden-form-submit-button" style="opacity: 0 !important; position: absolute !important; pointer-events: none !important" value="default" aria-hidden="true" tabindex="-1">Suivant</button>
<input type="hidden" name="state" value="hKFo2SBMNElHQ2w0ekNxVFpxa1ZmRnNoV2RMRWxLVTRfTE16TaFur3VuaXZlcnNhbC1sb2dpbqN0aWTZIEZTQUpidk94V2JjdTBoNEpTZHd3azBkcGVscnJwdElio2NpZNkga3NOV29pVDBmZ1lodGdYTDFZMm5sNDFPTW0xR3B0SXo">
<div class="cd3161405 cdece75f8">
<div class="c1213aeac">
<div class="input-wrapper _input-wrapper">
<div class="cc9cdc85a c65458272 text cd8a1eb40 ulp-field caa4c1d4a c409f0fe9" data-action-text="" data-alternate-action-text="">
<label aria-hidden="true" class="cd61d9fdd cb55ce585 c6d26c680" for="username">Email/Mobile</label>
<input class="input c59b33b40 c6b7d335a" inputmode="text" name="username" id="username" type="text" aria-label="Email/Mobile" value="" required="" autocomplete="username" autocapitalize="none" spellcheck="false">
<div data-lastpass-icon-root="" style="position: relative !important; height: 0px !important; width: 0px !important; float: left !important;"></div></div>
<div id="error-cs-username-required" class="ulp-error-info aria-error-check" data-ulp-validation-function="ulpRequiredFunction" data-ulp-validation-event-listeners="blur,change,input,focus" data-ulp-validation-target="username">Le nom d'utilisateur est requis</div>
<div id="error-cs-pattern-mismatch" class="ulp-error-info aria-error-check" data-ulp-validation-function="ulpPatternCheckFunction" data-ulp-validation-event-listeners="blur,change,input,focus" data-ulp-validation-target="username">Le nom d'utilisateur comporte des caractères non valides.</div>
</div>
</div>
</div>
<input class="hide" type="password" autocomplete="off" tabindex="-1" aria-hidden="true">
<input type="hidden" id="js-available" name="js-available" value="true">
<input type="hidden" id="webauthn-available" name="webauthn-available" value="true">
<input type="hidden" id="is-brave" name="is-brave" value="false">
<input type="hidden" id="webauthn-platform-available" name="webauthn-platform-available" value="true">
<input type="checkbox" name="rememberMe" id="rememberMe" class="input-remember-me" style="position: relative; bottom: -4px; width: 16px; height: 18px;"><label class="label-remember-me" for="rememberMe">Se souvenir de moi</label><i aria-hidden="true" class="tooltip-icon" data-toggle="tooltip" data-original-title="L'option Remember Me stocke les informations personnelles de l'utilisateur, ne cochez pas cette case s'il s'agit d'un appareil public." data-placement="right" style="background-image: url("https://ciamidpnext.s3.eu-central-1.amazonaws.com/miscellaneous/Remembeme.svg"); background-position: 1px 0px; height: 16px; width: 16px; display: inline-block; background-repeat: no-repeat; margin-left: 5px; bottom: -2px; position: relative;"></i><div class="c962c3db7">
<button type="submit" name="action" value="default" class="c374f5b8a c1085a438 ccdf87e4e cc02a3617 _button-login-id" data-action-button-primary="true">Suivant</button>
</div>
</form>
The client side does not matter, you need a server which does not give the same response when called twice.
@sesa501225 i need the form to be able to build a test case like org.htmlunit.html.HtmlForm2Test.inputTypeSubmitWithFormMethod(). During the test i can check the request count like assertEquals(2, getMockWebConnection().getRequestCount());
Having such tests enables me to run the tests with real browsers (selenium) and with the HtmlUnit driver to make sure the behaviour is identical.
@sesa501225 so far i see a form without an action attribute - means the form post request goes to the same url as the current page
But where is the js you are talking about in this game?
@sesa501225 you are using this version of the htmlunit driver?
https://github.com/SeleniumHQ/htmlunit-driver/blob/master/docs/compatibility.md
I just saw that I mixed htmlunit (you) and htmlunit-driver (selenium). I never updated htmlunit. And I cannot update it easily, htmlunit-driver is using old package "com.gargoylesoftware.htmlunit" in its code.
Thanks to your previous message, I have to use htmlunit3-driver instead of htmlunit-driver.
Still same behavior, a little worse even, before I could enable Javascript just after password page webWindowContentChanged event. Now I need to postpone it to just before password submit call.
And I cannot update it easily, htmlunit-driver is using old package "com.gargoylesoftware.htmlunit" in its code.
Yes expected, but the htmlunit-driver is only a thin wrapper, therefore i have to test/fix that in htmlunit. Making a fixed jdk8 driver is a later story.
but at first i have to construct a test case that reproduces your problem So far i did this:
but this works exactly as expected. I need to know what the JS stuff in your case does....
It seems to be a different issue now. I put a breakpoint in HttpPost and it is now called once (and not twice). However I have javascript error, and even with
newWebClient.getOptions().setThrowExceptionOnScriptError(false);
newWebClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
The webWindowContentChanged is never called for password page.
org.htmlunit.ScriptException: missing ; before statement (script in [xxxx] from (482, 13) to (659, 14)#569)
at org.htmlunit.javascript.JavaScriptEngine$HtmlUnitCompileContextAction.run(JavaScriptEngine.java:887)
at org.htmlunit.corejs.javascript.Context.call(Context.java:627)
at org.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:448)
at org.htmlunit.javascript.HtmlUnitContextFactory.callSecured(HtmlUnitContextFactory.java:312)
at org.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:722)
at org.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:751)
at org.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:948)
at org.htmlunit.html.ScriptElementSupport.executeInlineScriptIfNeeded(ScriptElementSupport.java:347)
at org.htmlunit.html.ScriptElementSupport.executeScriptIfNeeded(ScriptElementSupport.java:213)
at org.htmlunit.html.ScriptElementSupport$1.execute(ScriptElementSupport.java:112)
at org.htmlunit.html.ScriptElementSupport.onAllChildrenAddedToPage(ScriptElementSupport.java:131)
at org.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:216)
at org.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:514)
at org.htmlunit.cyberneko.xerces.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:283)
at org.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:460)
at org.htmlunit.cyberneko.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1236)
at org.htmlunit.cyberneko.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1180)
at org.htmlunit.cyberneko.filters.DefaultFilter.endElement(DefaultFilter.java:168)
at org.htmlunit.cyberneko.filters.NamespaceBinder.endElement(NamespaceBinder.java:265)
at org.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3176)
at org.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
at org.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:914)
at org.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:336)
at org.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:294)
at org.htmlunit.cyberneko.xerces.parsers.AbstractXMLDocumentParser.parse(AbstractXMLDocumentParser.java:79)
at org.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.parse(HtmlUnitNekoDOMBuilder.java:757)
at org.htmlunit.html.parser.neko.HtmlUnitNekoHtmlParser.parse(HtmlUnitNekoHtmlParser.java:196)
at org.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:300)
at org.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:219)
at org.htmlunit.WebClient.loadWebResponseInto(WebClient.java:681)
at org.htmlunit.WebClient.loadDownloadedResponses(WebClient.java:2708)
at org.htmlunit.html.DomElement.click(DomElement.java:1175)
at org.htmlunit.html.DomElement.click(DomElement.java:1096)
at org.openqa.selenium.htmlunit.HtmlUnitWebElement.submitForm(HtmlUnitWebElement.java:178)
at org.openqa.selenium.htmlunit.HtmlUnitWebElement.submitImpl(HtmlUnitWebElement.java:132)
at org.openqa.selenium.htmlunit.HtmlUnitDriver.lambda$runAsync$0(HtmlUnitDriver.java:351)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.htmlunit.corejs.javascript.EvaluatorException: missing ; before statement (script in [xxxx] from (482, 13) to (659, 14)#569)
at org.htmlunit.javascript.HtmlUnitContextFactory$HtmlUnitErrorReporter.error(HtmlUnitContextFactory.java:394)
at org.htmlunit.corejs.javascript.Parser.addError(Parser.java:247)
at org.htmlunit.corejs.javascript.Parser.reportError(Parser.java:327)
at org.htmlunit.corejs.javascript.Parser.reportError(Parser.java:319)
at org.htmlunit.corejs.javascript.Parser.reportError(Parser.java:315)
at org.htmlunit.corejs.javascript.Parser.autoInsertSemicolon(Parser.java:1451)
at org.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:1426)
at org.htmlunit.corejs.javascript.Parser.statement(Parser.java:1261)
at org.htmlunit.corejs.javascript.Parser.parse(Parser.java:636)
at org.htmlunit.corejs.javascript.Parser.parse(Parser.java:564)
at org.htmlunit.corejs.javascript.Context.parse(Context.java:2643)
at org.htmlunit.corejs.javascript.Context.compileImpl(Context.java:2575)
at org.htmlunit.corejs.javascript.Context.compileString(Context.java:1518)
at org.htmlunit.javascript.HtmlUnitContextFactory$TimeoutContext.compileString(HtmlUnitContextFactory.java:195)
at org.htmlunit.corejs.javascript.Context.compileString(Context.java:1506)
at org.htmlunit.javascript.JavaScriptEngine$HtmlUnitCompileContextAction.run(JavaScriptEngine.java:880)
... 38 common frames omitted
17:12:57.776 [PrivateDownloadTest.testPrivateDownloadSQE] ERROR c.s.bsl.it.AbstractIntegrationTest - Unable to init
org.htmlunit.ScriptException: missing ; before statement (script in [xxxx] from (482, 13) to (659, 14)#569)
at org.htmlunit.javascript.JavaScriptEngine$HtmlUnitCompileContextAction.run(JavaScriptEngine.java:887)
at org.htmlunit.corejs.javascript.Context.call(Context.java:627)
at org.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:448)
at org.htmlunit.javascript.HtmlUnitContextFactory.callSecured(HtmlUnitContextFactory.java:312)
at org.htmlunit.javascript.JavaScriptEngine.compile(JavaScriptEngine.java:722)
at org.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:751)
at org.htmlunit.html.HtmlPage.executeJavaScript(HtmlPage.java:948)
at org.htmlunit.html.ScriptElementSupport.executeInlineScriptIfNeeded(ScriptElementSupport.java:347)
at org.htmlunit.html.ScriptElementSupport.executeScriptIfNeeded(ScriptElementSupport.java:213)
at org.htmlunit.html.ScriptElementSupport$1.execute(ScriptElementSupport.java:112)
at org.htmlunit.html.ScriptElementSupport.onAllChildrenAddedToPage(ScriptElementSupport.java:131)
at org.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:216)
at org.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:514)
at org.htmlunit.cyberneko.xerces.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:283)
at org.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:460)
at org.htmlunit.cyberneko.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1236)
at org.htmlunit.cyberneko.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1180)
at org.htmlunit.cyberneko.filters.DefaultFilter.endElement(DefaultFilter.java:168)
at org.htmlunit.cyberneko.filters.NamespaceBinder.endElement(NamespaceBinder.java:265)
at org.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3176)
at org.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
at org.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:914)
at org.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:336)
at org.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:294)
at org.htmlunit.cyberneko.xerces.parsers.AbstractXMLDocumentParser.parse(AbstractXMLDocumentParser.java:79)
at org.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.parse(HtmlUnitNekoDOMBuilder.java:757)
at org.htmlunit.html.parser.neko.HtmlUnitNekoHtmlParser.parse(HtmlUnitNekoHtmlParser.java:196)
at org.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:300)
at org.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:219)
at org.htmlunit.WebClient.loadWebResponseInto(WebClient.java:681)
at org.htmlunit.WebClient.loadDownloadedResponses(WebClient.java:2708)
at org.htmlunit.html.DomElement.click(DomElement.java:1175)
at org.htmlunit.html.DomElement.click(DomElement.java:1096)
at org.openqa.selenium.htmlunit.HtmlUnitWebElement.submitForm(HtmlUnitWebElement.java:178)
at org.openqa.selenium.htmlunit.HtmlUnitWebElement.submitImpl(HtmlUnitWebElement.java:132)
at org.openqa.selenium.htmlunit.HtmlUnitDriver.lambda$runAsync$0(HtmlUnitDriver.java:351)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.htmlunit.corejs.javascript.EvaluatorException: missing ; before statement (script in [xxxx] from (482, 13) to (659, 14)#569)
at org.htmlunit.javascript.HtmlUnitContextFactory$HtmlUnitErrorReporter.error(HtmlUnitContextFactory.java:394)
at org.htmlunit.corejs.javascript.Parser.addError(Parser.java:247)
at org.htmlunit.corejs.javascript.Parser.reportError(Parser.java:327)
at org.htmlunit.corejs.javascript.Parser.reportError(Parser.java:319)
at org.htmlunit.corejs.javascript.Parser.reportError(Parser.java:315)
at org.htmlunit.corejs.javascript.Parser.autoInsertSemicolon(Parser.java:1451)
at org.htmlunit.corejs.javascript.Parser.statementHelper(Parser.java:1426)
at org.htmlunit.corejs.javascript.Parser.statement(Parser.java:1261)
at org.htmlunit.corejs.javascript.Parser.parse(Parser.java:636)
at org.htmlunit.corejs.javascript.Parser.parse(Parser.java:564)
at org.htmlunit.corejs.javascript.Context.parse(Context.java:2643)
at org.htmlunit.corejs.javascript.Context.compileImpl(Context.java:2575)
at org.htmlunit.corejs.javascript.Context.compileString(Context.java:1518)
at org.htmlunit.javascript.HtmlUnitContextFactory$TimeoutContext.compileString(HtmlUnitContextFactory.java:195)
at org.htmlunit.corejs.javascript.Context.compileString(Context.java:1506)
at org.htmlunit.javascript.JavaScriptEngine$HtmlUnitCompileContextAction.run(JavaScriptEngine.java:880)
... 38 common frames omitted
@sesa501225 any chance to provide your code (e.g. using my private mail) to give me a chance to debug this?
and looks like your are using this for doing your business - maybe you can ask your boss for some sponsoring of my work....
did a first check with a real browser, submitting the login form (post) returns a 302 response and the browser redirects (get) to the new location