[BUG] Error trying to access binary data from EXPath HTTP response

[amclark42]

Describe the bug Sometimes I get an error “Underlying channel has been closed” when trying to do something with the response returned from http:send-request().

After a lot of testing, I’ve narrowed down the problem scenario:

1. Use the HTTP Client to send a request to a URL which produces binary content (e.g. application/json)
2. Store the HTTP response in a variable $a
3. Pass $a to a function b()
4. If b() succeeds, try to do something with the response in $a (storage, decoding, etc.)

Number 4 fails with the error.

I ran into this when I started updating a function library to be more DRY. My first version of an XQuery had a lot of duplicate code, but didn’t produce any errors. Once I started testing the response code in another function, I started hitting this error over and over again.

Expected behavior I expected that the binary response contained in $a would still be usable after parsing it with b().

To Reproduce

xquery version "3.1";

  module namespace t="http://exist-db.org/xquery/test";
(:  LIBRARIES  :)
  declare namespace test="http://exist-db.org/xquery/xqsuite";
(:  NAMESPACES  :)
  declare namespace http="http://expath.org/ns/http-client";
  declare namespace output="http://www.w3.org/2010/xslt-xquery-serialization";
  declare namespace util="http://exist-db.org/xquery/util";
  declare namespace xdb="http://exist-db.org/xquery/xmldb";

(:~
  Trying to reproduce "Underlying channel has been closed" error when decoding 
  xs:base64Binary JSON.
  
  @author Ash Clark
  @since 2023
 :)
 
(:  VARIABLES  :)
  
  declare variable $t:request-url := 
    "https://wwp.northeastern.edu/exist/restxq/reception/sources";
  declare variable $t:request-element :=
    <http:request method="GET" timeout="15">  </http:request>;


(:  FUNCTIONS  :)
  
  declare
    %test:setUp
  function t:setup() {
    xdb:create-collection("/db", "test_2023-08-25")
  };
  
  declare
    %test:tearDown
  function t:tearDown() {
    xdb:remove("/db/test_2023-08-25")
  };
  
  (:~
    Send a request to the test URL using the EXPath HTTP Client. Make sure that the 
    request returns a 200 response, with a JSON payload.
    
    @return the EXPath HTTP Client's response headers, in an XML element
   :)
  declare
    %test:name('Response headers are OK')
      %test:assertExists
      %test:assertXPath('$result/@status eq "200"')
      %test:assertXPath('$result/Q{http://expath.org/ns/http-client}header[@name eq "content-type"][contains(@value, "application/json")]')
  function t:json-response-1-headers() as item() {
    http:send-request($t:request-element, $t:request-url)[1]
  };
  
  (:~
    Send a request to the test URL. Make sure that the response body is of type 
    xs:base64Binary.
    
    @return the HTTP response body
   :)
  declare
    %test:name("JSON response body is xs:base64Binary")
      %test:assertXPath('$result instance of xs:base64Binary')
  function t:json-response-2-body() as item() {
    http:send-request($t:request-element, $t:request-url)[2]
  };
  
  (:~
    Send a request to the test URL. Make sure that the response body can be saved to 
    eXist as a binary JSON file.
    
    @return a string filepath to the stored resource (if storage doesn't work, an empty sequence)
   :)
  declare
    %test:name("JSON response body can be saved as binary")
      %test:assertExists
  function t:json-response-3-save() as xs:string? {
    let $responseBody := http:send-request($t:request-element, $t:request-url)[2]
    return
      xdb:store-as-binary('/db/test_2023-08-25', 'test.json', $responseBody)
  };
  
  (:~
    Send a request to the test URL. Make sure that the response body can be turned 
    into a JSON string with `util:binary-to-string()`.
    
    @return the response JSON as a string
   :)
  declare
    %test:name("JSON response body can be decoded without saving")
      %test:assertExists
  function t:json-response-4-decode() as xs:string? {
    let $responseBody := http:send-request($t:request-element, $t:request-url)[2]
    return
      util:binary-to-string($responseBody)
  };
  
  (:~
    Send a request to the test URL. Check that `@status` on the first item of the 
    response sequence is '200', and if so, turn the response body into a JSON string.
    
    @return if the response status was "200", the response JSON as a string (otherwise, nothing)
   :)
  declare
    %test:name("JSON response body can be decoded after testing the response status")
      %test:assertExists
  function t:json-response-6-test-status-first() as xs:string? {
    let $response := http:send-request($t:request-element, $t:request-url)
    let $status := $response[1]/@status/data(.)
    return
      if ( exists($response) and $status eq '200' ) then
        let $responseBody := $response[2]
        return util:binary-to-string($responseBody)
      else ()
  };
  
  (:~
    Send a request to the test URL. Use a function `t:get-response-status()` to test 
    the HTTP response status for a '200'. If so, turn the response body into a JSON 
    string.
    
    @return if the response status was "200", the response JSON as a string (otherwise, nothing)
   :)
  declare
    %test:name("JSON response body can be decoded after using a function to test the response")
      %test:assertExists
  function t:json-response-7-test-through-function() as xs:string? {
    let $response := http:send-request($t:request-element, $t:request-url)
    let $status := t:get-response-status($response)
    return
      if ( exists($response) and $status eq '200' ) then
        let $responseBody := $response[2]
        return util:binary-to-string($responseBody)
      else ()
  };
  
  (:~
    Send a request to a different API endpoint that should return HTML. Use a 
    function `t:get-response-status()` to test the HTTP response status for a '200'. 
    If so, return the response body.
    
    @return if the response status was "200", the response HTML (otherwise, nothing)
   :)
  declare
    %test:name("HTML response body can be accessed after using a function to test the response")
      %test:assertExists
  function t:response-html-8-content-type() as node()? {
    let $reqURL := 
      "https://wwp.northeastern.edu/exist/restxq/reception/sources/americancritreview"
    let $response := http:send-request($t:request-element, $reqURL)
    let $status := t:get-response-status($response)
    return
      if ( exists($response) and $status eq '200' ) then
        $response[2]
      else ()
  };
  
  (:~
    Send a request to an endpoint that should return HTML. Use a function 
    `t:get-response-status()` to test the HTTP response status for a '200'. If so, 
    normalize whitespace in the HTML to produce a string.
    
    @return if the response status was "200", the response HTML (otherwise, nothing)
   :)
  declare
    %test:name("HTML response body can be manipulated after using a function to test the response")
      %test:assertExists
  function t:response-html-9-cast-as-string() as item()? {
    let $reqURL := "https://wwp.northeastern.edu/exist/restxq/reception/sources/americancritreview"
    let $response := http:send-request($t:request-element, $reqURL)
    let $status := t:get-response-status($response)
    return
      if ( exists($response) and $status eq '200' ) then
        normalize-space($response[2])
      else ()
  };
  
  (:~
    Send a request to a URL that should produce plaintext. Use a function 
    `t:get-response-status()` to test the HTTP response status for a '200'. If so, 
    normalize whitespace on the response body.
    
    @return if the response status was "200", the response text in one line (otherwise, nothing)
   :)
  declare
    %test:name("Plain text response body can be manipulated after using a function to test the response")
      %test:assertExists
  function t:response-txt-10-test-through-function() as item()? {
    let $reqURL := "https://raw.githubusercontent.com/eXist-db/exist/develop/README"
    let $response := http:send-request($t:request-element, $reqURL)
    let $status := t:get-response-status($response)
    return
      if ( exists($response) and $status eq '200' ) then
        normalize-space($response[2])
      else ()
  };
  
(:  SUPPORT FUNCTION  :)
  
  (:~
    Given an HTTP response from the EXPath HTTP Client, return the HTTP status code. 
   :)
  declare function t:get-response-status($http-response as item()*) as xs:string? {
    if ( exists($http-response) ) then
      $http-response[1]/@status/data(.)
    else ()
  };

When I run the test suite above in v6.2.0, all tests pass except for t:json-response-7-test-through-function(), which errors out.

Context (please always complete the following information)

• Build: eXist v6.2.0
• Java: 1.8.0_292 (OpenJDK)
• OS: Mac 12.6.2

Additional context

• How is eXist-db installed? JAR
• Any custom changes in e.g. conf.xml?
  • Tweaks made to keep whitespace from being added/removed (no indent, preserve whitespace).
  • I tried changing the Binary Manager to see if that would fix the bug (it didn't). It's back on the default org.exist.util.io.FileFilterInputStreamCache now.

[line-o]

I did experience the same behaviour. There might be a fix for this that already landed in develop-6.x.x https://github.com/eXist-db/exist/pull/4662

[line-o]

@amclark42 would you be able to try if your tests do pass with the latest version of eXist 6?

[amclark42]

@line-o Yes, I can try! It'll probably be later today; I have a bunch of meetings and not a lot of dev time right now.

Thanks for linking the PR, by the way. It definitely seems related to the issue I'm seeing. I'm hopeful that this has already been fixed!

[adamretter]

@amclark42 I think you said that you are using v6.2.0 of eXist-db. If that is the case, then the fix I made (https://github.com/eXist-db/exist/pull/4662) that @line-o suggested is already present in 6.2.0, so that won't help you with this I am afraid!

It might be worth trying the HEAD of the develop-6.x.x branch anyway, as there are also some other fixes I made there that are not in v6.2.0 that might solve your issue (e.g. https://github.com/eXist-db/exist/pull/4978)

[amclark42]

@adamretter @line-o

Sorry, I’m not having any luck building a package from the repository. I can’t check the test suite against a newer version. Would someone else try?