vlib/net/http: http proxy support

[dmitescu]

I am adding support for HTTP(S) proxies.

I added a struct HttpProxy, which checks if the connection should be http or https, and then modified the Request struct to contain an http proxy + a boolean to describe if it should use the proxy, and the do_http and do_ssl methods to check if a proxy exists. If it should use a proxy, it uses the connection supplied by the proxy, instead of dialing the target host and port.

Structs & Functions:

• vlib/net/http/http_proxy.v - added:
  • HttpProxy - which is similar to a Request; it contains the following methods:
    • prepare - checks if there is an open connection or if the host is the same, and if no, resets the connection and opens another one
    • open_connection - opens a connections to the proxy specifying the target destination
    • reset_connection - resets the connection (resets all states + closes it)
    • create_plain_tcp - just dials tcp
    • build_proxy_headers - builds the headers + command to send to the proxy (includes basic authentication)
  • new_http_proxy - self explanatory, I think
  • ProxyConnLayer - a struct describing the the proxied connection; it's just a fd, which implements certain features
• vlib/net/http/backend_(nix|windows).c.v - added create_ssl_connection which is a method of the HttpProxy struct
• vlib/net/openssl/c.v - added C.BIO_get_fd and C.BIO_set_fd
• vlib/net/tcp.v - added tcp_conn_from_handle (might have to go, as it's not a real tcp connection)

As this is WIP, the following things I'd like to ask:

• :question: a proxy connection can be reused if it is for the same host (at least this is the behaviour I noticed from curl); since it would impact the performance, i had to keep the TcpConn as a member of the struct - which then has to be mutable, sooo I had to change the Request to be mutable as well; is this okay?
• :heavy_check_mark: ~~for TLS-encrypted proxies, a TcpConn is returned: which is not really correct, since it's just a fd and I made a connection struct for convenience; there are better ways of doing this~~
• :question: should the ProxyConnLayer be refactored, and all these features written in a separate submodule? in the future people might want to implement other proxies, such as SOCKS and whatnow; maybe it's too early to ask this

... and the following things are left to be done:

• :x: windows support
• :heavy_check_mark: ~~update env method~~
• :heavy_check_mark: ~~tests~~ as much as one can write tests for this
• :interrobang: ~~i guess there is no way to check if the fd was closed, but as far as I tried, the proxy support works fine; if that's really necessary, one would have to implement "fd_is_closed" for both Linux and Windows~~ doing that, i found it was pretty impossible with all ioctls and selects :)
• :heavy_check_mark: ~~disable the MD5 cipher, and remove printlns~~

So... after some feedback on the current status, I will implement tests and proceed with the support for Windows (at work, in my lunchbreaks). We can chat on Discord as well.

[spytheman]

Great work! I especially liked how adding proxy support on windows, actually removed stuff from vschannel.c 🥳 .

[spytheman]

(git pull --rebase origin master to avoid the unrelated CI failures)

[dmitescu]

pffft I apparently messed up the case when the proxy is TLS encrypted (wget doesn't even support such proxies, nor could I find such proxies working online). I'll solve it.

[dmitescu]

solved for Linux (e76d862), Windows coming up. This can be used also for other purposes (i.e: an IRC client - so then there wouldn't be if(use_ssl) everywhere)

[medvednikov]

Hi @dmitescu

Great work!

What's your discord username?

[ylluminate]

@dmitescu friendly ping 😊

[JalonSolov]

The code definitely needs updates to work with latest V.

[JalonSolov]

ping?

[ArtemkaKun]

@dmitescu Hi, is this still actual? Should somebody from the community rebase this PR and try to merge it (in case you don't have time/are not interested in it anymore)

[enghitalo]

Can this be merged with master????

[ArtemkaKun]

I think this is still a WIP + need to be ported on a newer V

[lost22git]

It’s been almost two years. This PR hasn’t been merged to master yet. 🤔

[JalonSolov]

Can't be merged until it is caught up to latest V. Also needs tested on Windows, etc.

[osascruz]

This seems to be quite useful ! from china I need a proxy under windows, need set https_proxy=127.0.0.1:8080 to be supported v up Updating V... fatal: unable to access 'https://github.com/vlang/v/': Unsupported proxy syntax in '127.0.0.1:1080 ': Malformed input to a URL function

[ArtemkaKun]

So this can be closed after #19606 was merged?

[medvednikov]

@ArtemkaKun from spy:

afaik not yet it still has code that is not in the merged version (https and vschannel related) but I have not read it in a while the merged PR has support for HTTP proxies, and it is a bit cleaner so I think that the rest can be added without much effort including the handling of the HTTP_PROXY env variable (if it is set)

@sibkod your help with these would be greatly appreciated! Thanks.

[sibkod]

@ArtemkaKun from spy:

afaik not yet it still has code that is not in the merged version (https and vschannel related) but I have not read it in a while the merged PR has support for HTTP proxies, and it is a bit cleaner so I think that the rest can be added without much effort including the handling of the HTTP_PROXY env variable (if it is set)

@sibkod your help with these would be greatly appreciated! Thanks.

I'm working on it now. Socks5 and https proxy support is in progress.

[medvednikov]

I think this can now be closed :)

Great work @sibkod !

[JalonSolov]

http proxy support was added in #19606