HTTP request service unreachable (recaptcha and twilio)

[isaachall77]

Describe the bug With some consistency, external API HTTP requests will fail with error "5057 ERROR_SERVICE_UNREACHABLE". I'd say just about 50% of requests fail. We first noticed this when we implemented Twilio SMS for user verification on the password reset module and in testing, we would only receive text messages about half the time we were supposed to. We spent an extensive amount of time troubleshooting and even opened a couple tickets with Twilio support and they walked us through their own troubleshooting steps. Their conclusion was that it must be an issue with the way the application is making HTTP requests. The error was easily reproducible in PWM, but when using something simple like CURL to make the API call (from the same server or a different one) it worked 100% of the time. We can find nothing that would indicate it is our firewall or anything else interfering.

Workaround We have worked around the Twilio issue by standing up a second web server on our PWM server, running on a loopback port with some custom CGI. So now instead of making an API call to Twilio from PWM, PWM makes an API call to it's own loopback address and the API call is proxied to Twilio via a simple CURL script. This has a 100% success rate over the past two and a half months. However, PWM still has consistent issues making calls to recaptcha. And we cannot implement the workaround here, as the URL of recaptcha cannot be changed like it could with SMS.

To Reproduce Enable recaptcha or presumably any SMS API service and try using those services periodically. Try resetting your password and sending a text message to yourself, or purposely causing intruder attempts to make the captcha appear.

Expected behavior For recaptcha to work consistently, and to be able to make API calls to Twilio with the native code.

Desktop (please complete the following information): All browsers, all OSes.

PWM Version 2.0.0 Build 2020-01-09

Relevant logs 2020-02-09T17:25:41Z, ERROR, util.CaptchaUtility, unexpected error during reCaptcha API execution: 5057 ERROR_SERVICE_UNREACHABLE (error while making http request: www.recaptcha.net:443 failed to respond) (stacktrace follows) password.pwm.error.PwmUnrecoverableException: 5057 ERROR_SERVICE_UNREACHABLE (error while making http request: www.recaptcha.net:443 failed to respond) at password.pwm.svc.httpclient.PwmHttpClient.makeRequest(PwmHttpClient.java:326) at password.pwm.util.CaptchaUtility.verifyReCaptcha(CaptchaUtility.java:137) at password.pwm.util.CaptchaUtility.verifyReCaptcha(CaptchaUtility.java:96) at password.pwm.http.servlet.ForgottenUsernameServlet.handleSearchRequest(ForgottenUsernameServlet.java:139) at password.pwm.http.servlet.ForgottenUsernameServlet.processAction(ForgottenUsernameServlet.java:117) at password.pwm.http.servlet.AbstractPwmServlet.handleRequest(AbstractPwmServlet.java:123) at password.pwm.http.servlet.AbstractPwmServlet.doPost(AbstractPwmServlet.java:73) at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at password.pwm.http.filter.AbstractPwmFilter$PwmFilterChain.doFilter(AbstractPwmFilter.java:153) at password.pwm.http.filter.SessionFilter.processFilter(SessionFilter.java:110) at password.pwm.http.filter.AbstractPwmFilter.doFilter(AbstractPwmFilter.java:97) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at password.pwm.http.filter.AbstractPwmFilter$PwmFilterChain.doFilter(AbstractPwmFilter.java:153) at password.pwm.http.filter.ApplicationModeFilter.processFilter(ApplicationModeFilter.java:82) at password.pwm.http.filter.AbstractPwmFilter.doFilter(AbstractPwmFilter.java:97) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at password.pwm.http.filter.AbstractPwmFilter$PwmFilterChain.doFilter(AbstractPwmFilter.java:153) at password.pwm.http.filter.ObsoleteUrlFilter.processFilter(ObsoleteUrlFilter.java:65) at password.pwm.http.filter.AbstractPwmFilter.doFilter(AbstractPwmFilter.java:97) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at password.pwm.http.filter.RequestInitializationFilter.initializeServletRequest(RequestInitializationFilter.java:244) at password.pwm.http.filter.RequestInitializationFilter.doFilter(RequestInitializationFilter.java:166) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:263) at password.pwm.http.filter.GZIPFilter.doFilter(GZIPFilter.java:79) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at password.pwm.http.filter.CookieManagementFilter.doFilter(CookieManagementFilter.java:77) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1579) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.http.NoHttpResponseException: www.recaptcha.net:443 failed to respond at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) at password.pwm.svc.httpclient.PwmHttpClient.executeRequest(PwmHttpClient.java:454) at password.pwm.svc.httpclient.PwmHttpClient.makeRequestImpl(PwmHttpClient.java:354) at password.pwm.svc.httpclient.PwmHttpClient.makeRequest(PwmHttpClient.java:322) ... 52 more

[bwarncke2]

same problem here. Recaptcha does not work reliable, About 50% of requests fail with : ERROR_SERVICE_UNREACHABLE (error while making http request: www.recaptcha.net:443 failed to respond)

[zwets]

Same problem here, same error message. Running latest PWM release. Any takers?

[tatarenko]

I have expirience of fixing it First i put recaptcha.validateUrl=http://127.0.0.1:8089/recaptcha/api/siteverify in "App Property Overrides".

Then i made nginx configuration for it:

server {
        listen 127.0.0.1:8089;

        access_log /usr/local/www/recaptchalogs/nginx_access.log main;
        error_log /usr/local/www/recaptchalogs/nginx_error.log warn ;

        charset utf-8;
        client_max_body_size 64m;
        ssi on;
        ssi_value_length 1024;
        gzip_static on;

        location / {
                resolver 8.8.8.8 ipv6=off;
                proxy_pass https://www.recaptcha.net;
                proxy_set_header Host www.recaptcha.net;
                proxy_buffer_size          128k;
                proxy_redirect off;
                proxy_buffering off;
       }
}

After this i've got different error: 2022-08-04T01:38:57Z, FATAL, servlet.AbstractPwmServlet, {ovaqO} unexpected error: 5032 ERROR_CAPTCHA_API_ERROR (unexpected error during reCaptcha API execution: 5057 ERROR_SERVICE_UNREACHABLE (error while making http request: Connection reset)) [127.0.0.1]

And after changing keepalive_timeout 60; to keepalive_timeout 0;

in my nginx.conf in http section, the error was gone.

[jrivard]

There is also a /potential/ fix in v2.0.3 release.