Using fetch for unix socket

[n9]

Is it possible to use fetch for unix socket (e.g. to call docker API)? If not, what is the best way to communicate via HTTP through unix socket?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[n9]

Unstale.

[kitsonk]

Browser fetch does not support sockets. Node fetch declined it because of the lack of browser support. I would be opposed to it too.

Otherwise socket support was added in #4176.

So, respectfully, declined.

[n9]

Thank you:)

[lucacasonato]

Background

A unix socket is really just an alternative underlying transport for the request. It could be considered a proxy for the request. For background, here is how other languages / frameworks implement this:

In Node you can specify a socketPath in the http.request options. This only works for unix domain sockets, not named pipes on windows. Docs: https://nodejs.org/api/http.html#http_http_request_url_options_callback

In Go you can specify a custom Transport for your in the http.Client. This transport has a Dial hook that you can use to dial any transport type that you want, for example unix. See example: https://gist.github.com/ulfurinn/45d94d8bcc99e0a10025#file-gistfile1-go-L28-L36

It seems that in Python you have to use this package: https://pypi.org/project/requests-unixsocket/. You use the regular request method, but you specify your url to fetch as something like this http+unix://%2Fvar%2Frun%2Fdocker.sock/info.

Proposal

I propose we add this as a non standard extension in the unstable Deno.HttpClient as follows:

Deno.HttpClient gets a new proxy property that can be used to configure a unix socket transport. This aligns with how reqwest will likely implement this (https://github.com/seanmonstar/reqwest/issues/39). Here is what the TS type would look like:

  interface CreateHttpClientOptions {
    /** A certificate authority to use when validating TLS certificates. Certificate data must be PEM encoded.
     */
    caData?: string;
+
+   /** A proxy to use for the requests made with this socket. */
+   proxy?: Deno.UnixConnectOptions;
  }

In the future this could be extended to also allow allow Deno.ConnectOptions to specify a TCP connection for proxying. Further in the future we could also allow the user to pass a rid to a StreamResource, or even a Deno.Reader & Deno.Writer here to proxy over an arbitrary stream. That is not part of this proposal though.

An example:

const client = Deno.createHttpClient({
  proxy: {
    transport: "unix",
    path: "/var/run/docker.sock"
  }
});
const res = await fetch("http://localhost/info", { client });

By treating the unix socket as a proxy we can avoid issues related to things like needing to specify a custom host header, or :authority in http/2. See https://github.com/nodejs/node/issues/32326.

The security considerations for this would be the same as Deno.connect({transport: "unix"}). For a http request over a unix proxy, allow-net would not be required. For https URLs we would additionally require the --allow-net flag because we need to do some DNS resolving to validate TLS certificates.

[danopia]

Hi, I'm also looking for HTTP over Unix.

I find the above usage of 'proxy' interesting because the term is already defined for http interactions; case in point, Deno.createHttpClient({ proxy: { url: "..." }}) apparently now exists with a different structure/usecase. Would it still make sense to add unix sockets to that key as an overload?

[danopia]

I've published a workaround as /x/socket_fetch which implements a small subset of HTTP/1.1 in Typescript, so that Deno's support for bare Unix sockets can be used to send basic HTTP requests to e.g. Docker Engine.

An example:

import { fetchUsing, UnixDialer } from "https://deno.land/x/[email protected]/mod.ts";

const dialer = new UnixDialer("/var/run/docker.sock");
const resp = await fetchUsing(dialer, "http://localhost/v1.24/images/json");

To run that example locally:

$ deno run --unstable --allow-{read,write}=/var/run/docker.sock https://deno.land/x/[email protected]/examples/dialer/unix_docker.ts
200
Headers {
  server: "Docker/20.10.17 (linux)",
  "transfer-encoding": "chunked",
  [... other headers ...]
}
[... the json payload from docker engine ...]

This module is very limited (no POST, no socket reuse, etc) but should work fine for some basic use-cases.

[loynoir]

Any news?

Cannot use some npm library which use unix socket fetch, like dockerode.

Related https://github.com/denoland/deno/issues/17910

[loynoir]

Workaround

import { fetch, Agent } from 'undici'

const resp = await fetch('http://localhost/version', {
  dispatcher: new Agent({
    connect: {
      socketPath: '/var/run/docker.sock'
    }
  })
})

console.log(await resp.text())

$ deno run --unstable -A ./reproduce.mjs
{"Platform":...

[loynoir]

Any news?

undici workaround now not work, after updated deno and undici.

[fyapy]

@loynoir use import { fetch, Agent } from 'npm:[email protected]'

[Macil]

@lucacasonato Instead of changing Deno's definition of fetch (and RequestInit etc) to take a new client option, another possibility would be to make the HttpClient instance have a fetch method which works like the global fetch function:

const client = Deno.createHttpClient({
  proxy: {
    transport: "unix",
    path: "/var/run/docker.sock"
  }
});
const res = await client.fetch("http://localhost/info");

This possibility would also make so any Deno API compatibility layers re-implementing Deno.createHttpClient() won't need to shim its environment's global fetch function to support a client option.

(This idea occurred to me when I saw a similar pattern in Cloudflare Workers' classic HTTP Service Bindings, where you make a request to (or through) a Cloudflare Worker by calling the fetch method on the worker object as if it was the global fetch function.)