sneako
`Mint.UnsafeProxy` connection not handled in `Finch.Conn.request/6`
Hi, I'd like to report an issue of using Finch with HTTP proxy.
Problem
Requests to plain HTTP (i.e.: non-HTTPS) URLs via proxy are handled in Mint.UnsafeProxy, but currently Finch.Conn.request/6 is not aware of such case, resulting in FunctionClauseError.
Reproduce
defmodule FinchProxyTest.App do
use Application
def start(_type, _args) do
children = [
{Finch, name: FinchProxyTest, pools: pool_options()}
]
Supervisor.start_link(children, strategy: :one_for_one)
end
def pool_options do
%{host: host, port: port} = System.get_env("http_proxy", "") |> URI.parse()
%{default: [protocol: :http1, conn_opts: [proxy: {:http, host, port, []}]]}
end
end
# suppose env `http_proxy` is set
FinchProxyTest.Client.request "http://localhost:8000"
Possible Solutions
-
Handle
UnsafeProxyconnections inFinch.HTTP1.Pool.request/5 -
Dynamic Pool Configs: allow users to pass in a callback instead of static pool configs (
fn (scheme, host, port) -> pool_config end, for example)
Hi @iwinux! Thanks for trying Finch.
I was under the impression that this should work already as long as you are passing the proxy options correctly in your pool config.
Could you please share the full error that you see?
There might also be something going on in your Client module so it would probably be useful to share that as well.
The full error message:
iex(1)> FinchProxyTest.Client.request "http://localhost:8000"
** (FunctionClauseError) no function clause matching in Mint.HTTP1.close/1
The following arguments were given to Mint.HTTP1.close/1:
# 1
%Mint.UnsafeProxy{
hostname: "localhost",
module: Mint.HTTP1,
port: 8000,
proxy_headers: [],
scheme: :http,
state: %Mint.HTTP1{
buffer: "",
host: "127.0.0.1",
mode: :passive,
port: 1235,
private: %{},
proxy_headers: [],
request: nil,
requests: {[], []},
scheme_as_string: "http",
socket: #Port<0.6>,
state: :open,
streaming_request: nil,
transport: Mint.Core.Transport.TCP
}
}
Attempted function clauses (showing 2 out of 2):
def close(%Mint.HTTP1{state: :open} = conn)
def close(%Mint.HTTP1{state: :closed} = conn)
(mint 1.4.0) Mint.HTTP1.close/1
(finch 0.9.0) lib/finch/http1/conn.ex:172: Finch.Conn.close/1
(finch 0.9.0) lib/finch/http1/conn.ex:139: Finch.Conn.request/6
(finch 0.9.0) lib/finch/http1/pool.ex:46: anonymous fn/8 in Finch.HTTP1.Pool.request/5
(nimble_pool 0.2.4) lib/nimble_pool.ex:266: NimblePool.checkout!/4
(finch 0.9.0) lib/finch/http1/pool.ex:39: Finch.HTTP1.Pool.request/5
(finch 0.9.0) lib/finch.ex:274: Finch.request/3
(finch_proxy_test 0.1.0) lib/client.ex:6: FinchProxyTest.Client.request/1
The Client module has only minimal code:
defmodule FinchProxyTest.Client do
alias Finch.Response
def request(url) do
{:ok, %Response{body: body}} =
Finch.build(:get, url) |> Finch.request(FinchProxyTest)
body
end
end
And the content of mix.exs is:
defmodule FinchProxyTest.MixProject do
use Mix.Project
def project do
[
app: :finch_proxy_test,
version: "0.1.0",
elixir: "~> 1.12",
start_permanent: Mix.env() == :prod,
deps: deps()
]
end
def application do
[
mod: {FinchProxyTest.App, []},
extra_applications: [:logger]
]
end
defp deps do
[
{:finch, "~> 0.9.0"}
]
end
end
Thanks! Now it is clear what is happening.
Mint.HTTP is able to figure out the correct module to use, but Mint.HTTP1 is not.
Here we are using Mint.HTTP and there is a comment explaining why, but it looks like we probably will need to do the same throughout the module, and probably in the HTTP2 Pool as well in order to fully support proxying.
[...] and probably in the HTTP2 Pool as well in order to fully support proxying.
Mint does not support proxying for HTTP/2 connections since it cannot be done efficiently without knowledge of the pool.
Thanks for chiming in Eric, I missed that! In this case, we can ignore HTTP2 and just update HTTP1.
Hi @ericmj,
Just curious, what must have to be done for Mint to support HTTP/2 via proxy?
Just curious, what must have to be done for Mint to support HTTP/2 via proxy?
I should clarify that Mint supports implementing proxying on top of Mint, but Mint does not implement itself. So proxying could be implemented, but Finch should implement it itself.
The reason is because there are lots of different options for the pool on how to handle proxied connections and requests when using HTTP/2 which I will try to explain.
Unlike a HTTP/1 proxy, a HTTP/2 proxy can handle multiple tunnelled connections over a single connection to the proxy, because each tunnel uses a single stream and you can of course have multiple streams. Additionally, the tunnelled connection doesn't necessarily have to be a HTTP/2 connection, the tunnel is opaque to the proxy server so it can just as well be HTTP/1 (or even a non HTTP protocol if you like to support that).
Mint's built-in proxy functionality only fits one "end server" connection inside the proxy connection, but HTTP/2 proxies are way more flexible than that. In fact, even Mint's UnsafeProxy is limited to one "end server" connection, even though the HTTP protocol lets you do requests to multiple servers over one connection: GET https://foo.com/path and GET https://bar.com/path.
Mint's proxy functionality should be seen as best effort given what the API allows us to do. If more full-fledged functionality is needed then it needs to be implemented on top of Mint with a higher-level API that allows you to handle connections transparently, like Finch.
So even though Mint does not provide it for you, you should be able to implement it with Mint and if there is anything missing in Mint that prevents you from doing that we should address it.
Furthermore, there is additional complexity setting up secure tunnels inside SSL connections (which most HTTP/2 servers require). The connection to the proxy server is over SSL, but if the tunnel requires SSL you need to set up an additional SSL connection inside an SSL connection. AFAIK Erlang's ssl module does not expose an API to do this so you would have to hack around it and use private APIs.
