CLI: proxy setting gets ignored for HTTPS URLs

[DarthGizka]

I'm currently hacking a simple proxy as a solution to Offline use of validator_cli.jar: prevent Internet access attempts and also as a means for speeding up the validation of resources with invalid URLs. However, I've hit a snag that really puts a monkey wrench into my cunning plan: the proxy only receives requests for HTTP URLs and it gets bypassed completely for all HTTPS URLs.

Inspection of ValidatorCli::main shows that the JSSE doesn't get told about the proxy. Fixing this may be as simple as adding the system properties https.proxyHost and https.proxyPort (see JSSE guide). However, the -auth code path may need some attention in this regard as well.

Steps to reproduce: activate some means of monitoring HTTP traffic ('fiddlable' proxy and/or local web server and/or Wireshark) and validate the following resource with an appropriate proxy setting:

<Patient xmlns="http://hl7.org/fhir">
	<meta>
		<profile value="http://zrbj.eu/StructureDefinition/UNKNOWN-PROFILE-1"/>
		<profile value="https://zrbj.eu/StructureDefinition/UNKNOWN-PROFILE-2"/>
	</meta>
</Patient>

The proxy will get hit only for the HTTP profile URL; the request(s) for the HTTPS profile URL will bypass the proxy and time out. I've used zrbj.eu instead of zrbj.example because the latter might be deemed unresolvable a priori in some networking stacks. localhost seems to be configured in Java to bypass proxy requests by default.

[dkipping-mio42]

Currently also running into this issue (or at least I think the HTTPS is causing this issue)

With activated -proxy and -auth parameters, I am receiving time out errors while connecting to the build server for loading the R5 extensions:

java.net.SocketTimeoutException: connect timed out
        at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
        at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
        at java.net.PlainSocketImpl.connect(Unknown Source)
        at java.net.SocksSocketImpl.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
        at sun.net.NetworkClient.doConnect(Unknown Source)
        at sun.net.www.http.HttpClient.openServer(Unknown Source)
        at sun.net.www.http.HttpClient.openServer(Unknown Source)
        at sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source)
        at sun.net.www.protocol.https.HttpsClient.New(Unknown Source)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at java.net.HttpURLConnection.getResponseCode(Unknown Source)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
        at org.hl7.fhir.utilities.SimpleHTTPClient.get(SimpleHTTPClient.java:149)
        at org.hl7.fhir.utilities.SimpleHTTPClient.get(SimpleHTTPClient.java:121)
        at org.hl7.fhir.utilities.npm.FilesystemPackageCacheManager.loadFromBuildServer(FilesystemPackageCacheManager.java:672)
        at org.hl7.fhir.utilities.npm.FilesystemPackageCacheManager.checkBuildLoaded(FilesystemPackageCacheManager.java:661)
        at org.hl7.fhir.utilities.npm.FilesystemPackageCacheManager.checkCurrency(FilesystemPackageCacheManager.java:642)
        at org.hl7.fhir.utilities.npm.FilesystemPackageCacheManager.loadPackage(FilesystemPackageCacheManager.java:500)
        at org.hl7.fhir.r5.conformance.R5ExtensionsLoader.loadR5Extensions(R5ExtensionsLoader.java:40)
        at org.hl7.fhir.validation.cli.services.ValidationService.initializeValidator(ValidationService.java:347)
        at org.hl7.fhir.validation.cli.services.ValidationService.initializeValidator(ValidationService.java:326)
        at org.hl7.fhir.validation.ValidatorCli.doValidation(ValidatorCli.java:259)
        at org.hl7.fhir.validation.ValidatorCli.main(ValidatorCli.java:168)

[dkipping-mio42]

Could the validator CLI not use the system proxy configuration of Java by default? So the proxy settings which are configured in the Java control panel? Or am I missing something here, which requires the configuration of proxy settings via separate parameters?

[dotasek]

At the moment, the -proxy parameter is parsed and passed through to the Java system properties:

Example:

-proxy localhost:8080

... is the equivalent of:

-Dhttp.proxyHost=localhost
-Dhttp.proxyPort=8080

As @DarthGizka mentions, you can also set them explicitly.

I can confirm that this seems to be working when I point to mitmproxy running on localhost.

BUT...

I'm currently investigating what happens when using the https equivalent (https.proxyHost and https.proxyPort). At a glance, @DarthGizka's observation appears correct; checking for an https profile doesn't appear to completely work with these settings.

mitmproxy log of Validating his example Patient resource:

I am seeing at least one https connection routed through the proxy, which is good, and doesn't happen when no https proxy is set. BUT I should see a request for https://zrbj.eu/StructureDefinition/UNKNOWN-PROFILE-2. Within my validator output there's a caught error:

Warning @ Patient.meta.profile[0] (line 1, col38): Profile reference 'https://zrbj.eu/StructureDefinition/UNKNOWN-PROFILE-2' has not been checked because it is unknown, and fetching it resulted in the error java.lang.NullPointerException: Parameter specified as non-null is null: method okhttp3.Credentials.basic, parameter username

I'm investigating why this might be happening.

[dotasek]

@DarthGizka I have a branch that appears to work with https proxy set:

https://github.com/hapifhir/org.hl7.fhir.core/tree/dotasek-https-proxy

Do you know how to build the validator-cli jar yourself to test if it would work for you?

[DarthGizka]

@dotasek Sorry, no can do ATM. Air-gapped at work because we're dealing with health data, and since I currently work pretty much 7 x 14-16 there isn't much else ... When things normalise again then I'll set up a build environment an my Java notebook that I built expressly for working with HAPI :-) and do a deep dive, but the way things are looking that probably won't be before September at the earliest. Sorry for not being able to contribute ATM.

[dotasek]

Implemented in https://github.com/hapifhir/org.hl7.fhir.core/pull/888