net4people
Confirmed block of default Snowflake broker in China, 2023-05-12 to 2023-05-15
Thanks for the news. We will keep an eye on it.
These are OONI charts relevant to Snowflake in China:
- https://explorer.ooni.org/chart/mat?probe_cc=CN&since=2023-04-22&until=2023-05-22&time_grain=day&axis_x=measurement_start_day&test_name=web_connectivity&domain=cdn.sstatic.net
- https://explorer.ooni.org/chart/mat?probe_cc=CN&since=2023-04-22&until=2023-05-22&time_grain=day&axis_x=measurement_start_day&test_name=torsf
See #197 for some suggestions for workarounds for when the default Snowflake rendezvous was temporarily blocked in Iran. The AMP cache rendezvous won't work unless you can find an unblocked Google front domain, but you can also find a list of possible alternative front domains there and instructions for activating them.
You may be able to get more information from the snowflake-client log, but you will have to enable it manually. Here are instructions from #131:
We would like to see snowflake-client logs from failed connections. This log provides more information (e.g. "unable to create broker channel") than the Tor log does ("Bootstrapped 10%"), but you need to take special steps to activate it. In Tor Browser desktop, edit the file Browser/TorBrowser/Data/Tor/torrc-defaults. Find the line that starts with
ClientTransportPlugin snowflakeand add this to the end of the line:
-log snowflake-client.log -log-to-state-dirThen, when you restart Tor Browser, you will find the log at:
- linux: Browser/TorBrowser/Data/Tor/pt_state/snowflake-client.log
- windows: Browser\TorBrowser\Data\Tor\pt_state\snowflake-client.log
- mac: ~/Library/Application Support/TorBrowser-Data/Tor/pt_state/snowflake-client.log (Use Go to Folder... in the Finder menu.)
I deleted the previous testing results, which is wrong. Seems my local ISP has other restrictions to Fastly IPs.
wrong results
Yes, from my testing: about 3 times of TLS client hellos containing the SNI cdn.sstatic.net to a Fastly CDN IP will cause a temporary packet drop to its 443 port.
It differs from what GFW has done for github.com or store.steampowered.com: the packet drop happens to any IPs outside China instead of a specific AS like Fastly.
Opened a thread on Tor forum: https://forum.torproject.net/t/snowflake-bridge-does-not-work-in-china-since-days-ago/7635 .
if you ping them directly the packet loss seems fine.
ping test should always go through because the SNI-based blocking usually only happens on TCP 443 port. You may use curl to perform a HTTPS request or netcat to probe the 443 port periodically during the Snowflake initialization process.
But when trying to connect with the builtin snowflake of tor browser it does trigger the blocking somehow.The Wireshark result shows TCP Retransmission or sometimes SYN,ACK error right after Client Key Exchange.
Does this blocking target on that Fastly CDN IP? ~~I mean the IP of snowflake-broker.torproject.net.global.prod.fastly.net.~~ Seems Tor will use front's IP instead of url's.
UPDATE: And, I figured out that it's the IP of the front domain: cdn.sstatic.net is blocked in my area. I tried changing it to another Fastly domain mirrors.fastly.net and it works fine.
In my area there're more limitations to Fastly CDN IPs:
- Almost all of them are blocked including IPs of
fastly.com,fastly.net,fastly.jsdelivr.netand of coursecdn.sstatic.net. - Two repeated HTTPS request with any same SNI (even including
www.gov.cn) to that IP will result in a temporary ban of that IP, even the ICMP won’t go through, requests without SNI won't trigger that. Interval between the two requests should below 60 seconds. The ban lasts for 180 seconds. The ban does not target on some of Fastly CDN IPs.
For 2, some may say it's Fastly applied some types of rate limit, but how could it be 180 seconds accidentally, 180 is very typical of GFW.
There was a paper published in January 2023 about detecting Snowflake connections. One of the authors is professor Cheng Guang (程光), who works on a lot of topics related to encrypted traffic analysis.
The Snowflake detection algorithm has two parts: a coarse-grained prefilter based on DNS queries; and a fine-grained machine learning classifier on DTLS handshake features. The DNS prefilter (Section 4.1) looks for DNS queries to both the default front domain cdn.sstatic.net and some number of default STUN servers. Source IP addresses that are caught in the prefilter then move on to the DTLS fingerprint classification (Section 4.2). This uses features (Table 1) extracted from the DTLS Client Hello and Server Hello:
| Feature | Client Hello | Server Hello |
|---|---|---|
| DTLS record version | ✓ | ✓ |
| Record length | ✓ | ✓ |
| Fragment offset | ✓ | ✓ |
| Fragment length | ✓ | ✓ |
| Cipher suite length | ✓ | ✓ |
| Cipher suites | ✓ | |
| Extension Length | ✓ | ✓ |
| Extensions | ✓ | ✓ |
| Cipher suit chosen | ✓ |
Figure 6 shows the relative importance of DTLS fingerprint features in their classifer:
| Feature | Importance | Comment |
|---|---|---|
| S_cipher_chosen_b'\xc0/' | 0.18 | 0xc02f = TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 |
| S_fragment_len | 0.12 | |
| C_cipher_suits_5 | 0.10 | |
| C_cipher_suitlen | 0.09 | |
| S_HP_len | 0.08 | do not know what this is |
| S_cipher_chosen_b'\xc0+' | 0.06 | 0xc02b = TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 |
| C_HP_len | 0.05 | do not know what this is |
| S_extensions_8 | 0.05 | |
| C_fragment_len | 0.04 | |
| S_extensions_3 | 0.04 |
Just because someone wrote a paper, does not mean that the technique they propose is actually used in practice. But it gives some ideas to think about.
- Two repeated HTTPS request with any same SNI (even including
www.gov.cn) to that IP will result in a temporary ban of that IP, even the ICMP won’t go through, requests without SNI won't trigger that. Interval between the two requests should below 60 seconds. The ban lasts for 180 seconds.
That may be a very relevant fact, that two repeated HTTPS requests are required. Back in December 2022, Tor Browser 12.0 added a second Snowflake bridge. Unfortunately, due to technical limitations in the integration with Tor, having two bridges means that the Snowflake client sends two broker rendezvous messages (to the configured front domain) when it starts up. See the notes here, which basically describe how it works now:
- then there can be multiple snowflake bridge lines in torrc, each with different bridge fingerprints. load balancing will come from the tor client's local random selection.
- there is a "thundering herd" concern with the way tor currently uses multiple bridge lines. tor will attempt to connect to all of them at once, and keep using only one of them. This means N broker transactions and N STUN exchanges, and possibly N−1 proxies held idle.
- maybe it's not such a big problem, if N is not too large
- an alternative would be a modification to tor where it shuffles its bridge list, and tries only one at a time
- another alternative is to write the torrc file dynamically: choose a random bridge line, then write torrc containing only that one randomly selected bridge line
If that is the case, that simultaneous broker rendezvous messages is a feature used to detect Snowflake connections, you may be able to work around it by disabling one of the bridges in Tor Browser. In the bridges configuration, if you enabled Snowflake as a built-in bridge, it should show two bridges:
snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 ...(✈️☎️👑🏈)snowflake 192.0.2.4:80 8838024498816A039FCBBAB14E6F40A0843051FA ...(🍕🏡🐠🚀)
You can copy and paste one of the bridge lines as a manually entered bridge, rather than a built-in bridge. (Make sure you copy the entire bridge line, everything from snowflake ... utls-imitate=hellorandomizedalpn.) The bridge with fingerprint 8838024498816A039FCBBAB14E6F40A0843051FA is currently less loaded, so it's better to use that one.
Current versions of Orbot (before v17.0) also only have one Snowflake bridge configured; you could try Snowflake in Orbot and see if it works better than Tor Browser desktop.
Issue #40578 in Tor ("Let bridge users choose to only reach their first working bridge") is to fix the multiple rendezvous problem. It has a patch, but no action beyond that so far.
having two bridges means that the Snowflake client sends two broker rendezvous messages
Basically Tor can do this with only one Client Hello by reusing same HTTPS connection. I'm wondering will Tor make two separated TLS Client Hello which will trigger the 180s ban or not.
EDIT: Yes. From the Wireshark records, Tor will indeed send Client Hello twice. Since there are only two Snowflake bridges, this twice is enough, Tor can bootstrap to 100%, but not if there is one more, because the opportunity has been exhausted, and the GFW has already started blocking the IP.