Using `HEAD` request fails with "`Failed to open/read local data from file/application`" when used on `tiny_httpd` directory server.

[zoj613]

I have a function where I attempt to get the size in bytes of a resource using the HEAD request like so:

	module C = Ezcurl
  	exception Request_failed of int * string
  	let raise_error (code, s) = raise (Request_failed (Curl.int_of_curlCode code, s))
  	let fold_result = Result.fold ~error:raise_error ~ok:Fun.id

    let get t key =
      let tries = t.tries and client = t.client and config = t.config in
      let url = t.base_url ^ key in
      let+ res = C.get ~tries ~client ~config ~url () in
      match fold_result res with
      | {code; body; _} when code = 200 -> body
      | {code; body; _} -> raise (Request_failed (code, body))

    let size t key =  
      let tries = t.tries and client = t.client and config = t.config in
      let url = t.base_url ^ key in
      let type' = if String.ends_with ~suffix:".json" key then "json" else "octet-stream" in
      let headers = [("Content-Type", "application/" ^ type')] in
      let* res = C.http ~headers ~tries ~client ~config ~url ~meth:HEAD () in
      match res with
      | Error e -> raise_error e
      | Ok {code; _} when code = 404 -> Deferred.return 0
      | Ok {headers; code; _} when code = 200 ->
        begin match List.assoc_opt "content-length" headers with
        | Some "0" -> Deferred.return 0
        | Some l -> Deferred.return @@ int_of_string l
        | None ->
          begin try
          get t key >>| String.length with
          | Request_failed (404, _) -> Deferred.return 0 end
        end
      | Ok {code; body; _} -> raise (Request_failed (code, body))

On the server side, the handlers for uploading and downloading look like (using a tiny_httpd local directory server):

    (* HEAD request handler *)
    S.add_route_handler server ~meth:`HEAD S.Route.rest_of_path_urlencoded (fun path _ ->
      let headers = [("Content-Type", if String.ends_with ~suffix:".json" path then "application/json" else "application/octet-stream")] in
      let fspath = Filename.concat dir path in
      let headers = match In_channel.(with_open_gen [Open_rdonly] 0o700 fspath length) with
        | exception Sys_error _ -> ("Content-Length", "0") :: headers
        | l -> ("Content-Length", Int64.to_string l) :: headers
      in
      let r = S.Response.make_raw ~code:200 "" in
      S.Response.update_headers (List.append headers) r
    );
    (* GET request handler *)
    S.add_route_handler server ~meth:`GET S.Route.rest_of_path_urlencoded (fun path _ ->
      let fspath = Filename.concat dir path in
      match In_channel.(with_open_gen [Open_rdonly] 0o700 fspath input_all) with
      | exception Sys_error _ -> S.Response.make_raw ~code:404 (Printf.sprintf "%s not found" path)
      | s ->
        let headers =
          [("Content-Length", Int.to_string (String.length s))
          ;("Content-Type",
            if String.ends_with ~suffix:".json" path
            then "application/json"
            else "application/octet-stream")]
        in
        S.Response.make_raw ~headers ~code:200 s
    );
     (* POST request handler *)
    S.add_route_handler server ~meth:`POST S.Route.rest_of_path_urlencoded (fun path req ->
      let write oc = Out_channel.(output_string oc req.S.Request.body; flush oc) in
      let fspath = Filename.concat dir path in
      Zarr.Util.create_parent_dir fspath 0o700;
      let f = [Open_wronly; Open_trunc; Open_creat] in
      match Out_channel.(with_open_gen f 0o700 fspath write) with
      | exception Sys_error e -> S.Response.fail ~code:500 "Upload error: %s" e
      | () ->
        let opt = List.assoc_opt "content-type" req.headers in
        let content_type = Option.fold ~none:"application/octet-stream" ~some:Fun.id opt in
        let headers = [("content-type", content_type); ("Connection", "close")] in
        S.Response.make_string ~headers (Ok (Printf.sprintf "%s created" path))
    );

If I try to call the set function on a server with the handlers as describe above, CURL fails with error code 26 and a message "Failed to open/read local data from file/application". However If I implement set using a GET method like below, there are no exceptions thrown.

    let size t key = try get t key >>| String.length with
      | Request_failed (404, _) -> Deferred.return 0

Looking at the stack trace on my terminal it shows that the failure is caused by the call to C.http using the HEAD method. I have no idea why this is the case and i'm hoping you can shed some light on how to resolve this issue. Any ideas?

[c-cube]

I'm not sure what the problem is. I tried locally with:

• server side:

  Server.add_route_handler ~meth:`HEAD server
    Route.(exact "head" @/ return)
    (fun _req ->
      Response.make_void ~code:200 ~headers:[ "x-hello", "world" ] ());

• client side using Ezcurl (sync):

# module C = Ezcurl;;
# C.http ~meth:HEAD ~url:"http://localhost:8085/head" () ;;
- : (string C.response, Curl.curlCode * string) result =
Ok
 {Ezcurl_core.code = 200;
  headers = [("content-length", "0"); ("x-hello", "world")]; body = "";
  info = {Ezcurl_core.ri_response_time = 0.000878; ri_redirect_count = 0}}

[zoj613]

I read online that this sort of error may be because of an incorrect path or permission problem. I note that in your handler you use Route.(exact "head" @/ return) instead of S.Route.rest_of_path_urlencoded. Is there ready any reason for not using rest_of_path if one plans to handle arbitrary resource paths?

[c-cube]

It might be a subtle discrepancy somewhere between the actual path and the expected URL encoded one, for sure. If you pass a/b/c as a URL path, it won't match rest_of_path_urlencoded. Maybe try with a path without / ?

[zoj613]

It might be a subtle discrepancy somewhere between the actual path and the expected URL encoded one, for sure. If you pass a/b/c as a URL path, it won't match rest_of_path_urlencoded. Maybe try with a path without / ?

I think this has something to do with the resource maybe not being written to the correct path or the GET/HEAD method reading from the wrong path. Is there any other way I can write S.Route.rest_of_path_urlencoded in the HEAD and POST handlers such that it works for an arbitrary path? I'd like to try that just to see if the issue is the use of S.Route.rest_of_path_urlencoded

[c-cube]

Again, are you trying with a full file path (containing '/')? If so, you need to URL encode it first.

[zoj613]

let url = t.base_url ^ key is the full url where key is the full path after tha base_url (base_url being something like localhost/. Doesn't S.Route.rest_of_path_urlencoded URL encode the path part of url? I'm not sure what I may be doing wrong here.

[c-cube]

You mean URL decode?

It'd be helpful to know what URL exactly is sent by size (I don't see any URL encoding there), which URL is received by tiny_httpd, and which URL does the handler try to use.

[zoj613]

An example URL sent by size would be something like 127.0.0.1:8080/path/to/a/jsonfile.json or 127.0.0.1:8080/path/to/a/resource/with/plain/bytes. I have no idea what URL is received by tiny_httpd. I assumed it is the same as the one sent by the request. Is this a correct assumption? I have no clear idea how tiny_httpd handlers try to use the URL. I do not think the Tiny_httpd.Route.* functions are well documented so I have no confidence that I used Tiny_httpd.Route.rest_of_path_urlencoded correctly.

[c-cube]

You can always print the path from your handler, to see if it's actually invoked and with what?