Rehash retrieval proxies / multiple transports / inconsistent source attack protection (aka "smart retrievals")

[aesedepece]

This is a rehash of #1523

Enables using proxies (HTTP, HTTPS, SOCKS4 and SOCKS5) when performing data retrieval. This allows retrieving data sources through different transports so as to ensure that the data sources are consistent and we are taking as small of a risk as possible when committing to specially crafted data requests that may be potentially ill-intended.

Just add this to the [connections] section of witnet.toml to proxy your retrievals through 3 different open proxies:

retrieval_proxies = [
        "socks5://181.102.35.195:1080",
        "socks5://8.210.60.118:1080",
        "socks5://47.114.8.133:1080"
]

These addresses have been taken from this public list of proxies.

Close #1274 Close #1511

[aesedepece]

This is still a draft because tests need yet to be fixed, and I want to improve some other parts.

[aesedepece]

Thanks @tmpolaczyk for all your suggestions. I'll try to address them all today.

[tmpolaczyk]

I just found a very funny issue, this PR breaks RNG requests because the nodes commit a "ModeTie" error instead of a random value.

[tmpolaczyk]

When trying to use a public socks5 proxy, I get this error when trying a request with https sources (didn't test http):

[2022-08-11T11:36:19Z WARN  isahc::handler] request completed with error: the server certificate could not be validated

witnet.toml:

proxies = [
        "socks5://72.217.216.239:4145",
]

I tried other proxies and I got either the same error or a "retrive timeout" error, so not sure if I had bad luck choosing the proxies or it doesn't work as expected for some reason.

[aesedepece]

I tried other proxies and I got either the same error or a "retrive timeout" error, so not sure if I had bad luck choosing the proxies or it doesn't work as expected for some reason.

I tried a few proxy addresses as well and found out that most of those that you can find out there are garbage, regardless of being SOCKS or HTTP.

The only one that worked like a charm was the SOCKS5 interface on port 9150 from my own Tor Browser. This added like 1-2s of latency but still all the retrievals were successful (except one API that gave me 403 because they're behind a WAF or something).

[tmpolaczyk]

I tried a few proxy addresses as well and found out that most of those that you can find out there are garbage, regardless of being SOCKS or HTTP.

That seems to be true. I finally got it working by running a local socks server.

[tmpolaczyk]

Proxy URLs are validated at startup, but there is no connection attempt to ensure that they work.

There is no easy way to manually check that proxies work:

• The witnet toolkit binary cannot be used to test proxies because it doesn't read the witnet.toml file.
• The witnet node sendRequest command reads the witnet.toml file but ignores the proxy options.
• Sheikah connected to the wallet reads the witnet.toml and uses the witnessing config, but in case of timeout there is some kind of bug and sheikah displays the loading icon instead of the error. After waiting more than one minute it finally displays the error "Source looks inconsistent when...", but that error also doesn't mention which proxy failed.

So unless I missed something there is no way to check if the configured proxies work, it would be nice to have something like a test-request CLI command.

In the node, when one proxy fails with a timeout error, this is the only log:

[2022-09-30T08:27:10Z ERROR witnet_node::actors::chain_manager::mining] Couldn't resolve rad request 86920ddf5d7066b1db629066ab50118855c05bf7fbf3d4ba53e1154c3ca02b43: Timeout during retrieval phase

It would be nice if the error message could mention the URL of the proxy that failed.

[aesedepece]

Proxy URLs are validated at startup, but there is no connection attempt to ensure that they work.

In my mind I had implemented full validation (by querying some well-known URL). But it looks like that happened only in my mind. Would you be OK if we go with this as it is and implement that or a CLI method (or both) in a further PR?

[tmpolaczyk]

Would you be OK if we go with this as it is and implement that or a CLI method (or both) in a further PR?

Yes, it could be a separate PR. I suggested it here in case that makes us rethink the configuration parameters or something, but the basic structure of the witnessing params looks solid enough.

Then if I haven't missed any breaking changes this is ready to merge.