bbs
bbs copied to clipboard
net4people
Drop in Snowflake users from Russia and evidence of blocking starting 2024-11
We've noticed a significant drop in Snowflake users from Russia, starting earlier this month, along with some vantage point testing that suggests a different kind of block than what we've seen before.
At first we thought this was due to several recent certificate renewals of Fastly front domains that were used for Snowflake, but we've since ruled out blocking of rendezvous channel. The timing of the drop in users also doesn't quite match up with the domain renewals.
From our vantage point tests, it does not look like the DTLS fingerprinting of Snowflake that happened in Russia in 2021. The DTLS handshake completes and several bytes of application data are received from the proxy before the client suddenly stops receiving data. This doesn't rule out fingerprinting. There are a few proxies that do not appear to be blocked and Tor can be fully bootstrapped through these proxies, but we haven't noticed any difference in the DTLS fingerprint between working proxies and proxies that go stale. It's also possible that our vantage point tests aren't offering an accurate view of why there is a drop in users.
Related Links:
- Snowflake GitLab issue: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40407
- Full analysis of previous DTLS fingerprinting: https://gitlab.torproject.org/tpo/anti-censorship/pluggable-transports/snowflake/-/issues/40014#note_2765061
- ntc.party topic: https://ntc.party/t/drop-in-snowflake-users-from-russia-and-evidence-of-blocking-starting-2024-11/13083
on my Android Orbot Snowflake Relay I saw a recent sudden increase in twice as much russian connections as usual per day. I already thought, that something must be going on. Just spread that more people start installing the snowflake browser extension.
I find the following result interesting. Because earlier this year in January I experience the similiar situation in China.
What's weird is that data channel connections to proxies appear to be opening successfully, and some data is being sent bidirectionally before the connections become stale. From one of the snowflake probe logs (edited to remove extraneous messages)
Here is the post I made at that time:
https://github.com/net4people/bbs/issues/325
and @wkrp wrote this in that post:
The anti-censorship team looked into this report a little. The cause is uncertain. The logs show STUN, rendezvous, and DTLS connection establishment working correctly, but apparently not much data is transferred over the DTLS connection before it is closed.
My experience at that time was that the snowflake bootstrap could not even finish most of time. But in rare case they did finish and wireshark would start to show some DTLS Application Data transfer. But no websites could be opened. And in even rarer case website could work, It would be very slow and the connection would only last for a really short time (no more than 1 to 3 minutes). Then the connection would be blocked and wireshark would show Encrypted Alert messages.
The problem lasted for more than one month then gone, We couldn't figure out the exact reason back then. At that time we even couldn't figure out the width of the blocking. I am not sure whether that case is related with this one. But still maybe the experience is kinda similar?
I've written a patch for snowflake to more easily test Snowflake connections from vantage points. Manual WebRTC signalling can be used to test client connections to a specific Snowflake proxy under your control that also has manual signalling enabled. Hopefully this is useful to anyone who wants to try out theories on how it is being blocked or test fixes.
You will need the following patch for both the client and the proxy: https://gitlab.torproject.org/cohosh/snowflake/-/commit/9523d9bbed0baf44e8779c1c78bb7ebd4e501a8a
Proxy setup
To run the proxy, you will need 2 terminals:
- Terminal A: Run the proxy with
./proxy --copy-paste --verbose --unsafe-logging - Terminal B: Run
cat > signal
Client setup
To run the client, you will need 3 terminals, and the following torrc file:
UseBridges 1
DataDirectory datadir
ClientTransportPlugin snowflake exec ./client -copy-paste -log snowflake.log --unsafe-logging
Bridge snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 ice=stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 utls-imitate=hellorandomizedalpn
SocksPort auto
- Terminal A: Run the client with
tor -f torrc - Terminal B: Run
tail -f snowflake.log - Terminal C: Run
cat > signal
Manual signalling
Once the client has started, you will see something like the following output in the client's Terminal B.
2024/11/20 00:45:58 Please Copy & Paste the following to the peer:
2024/11/20 00:45:58 ----------------
2024/11/20 00:45:58
{"type":"offer","sdp":"v=0\r\no=- 7792309864421436583 1732063557 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS*\r\na=fingerprint:sha-256 82:33:BE:0E:A0:DC:FC:24:85:B5:7B:D5:9D:64:32:C9:A4:2F:56:B0:A6:47:35:B0:A4:4F:5D:96:E9:A6:7F:DA\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:actpass\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:vggBpjGDVijaxuQY\r\na=ice-pwd:skiZVUcgjPZpSyENHMHFBtRGUszeInMU\r\na=candidate:1408731006 1 udp 1694498815 XX.XX.XX.XX 40705 typ srflx raddr 0.0.0.0 rport 40705\r\na=end-of-candidates\r\n"}
2024/11/20 00:45:58 ----------------
Copy that, and paste it into the proxy's Terminal B. Soon after, you should see the following output in the proxy's Terminal A:
2024/11/20 00:46:05 Please Copy & Paste the following to the peer:
2024/11/20 00:46:05 ----------------
2024/11/20 00:46:05
{"type":"answer","sdp":"v=0\r\no=- 415248651781399192 1732218246 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic:WMS*\r\na=fingerprint:sha-256 25:C6:51:5D:35:61:F6:21:40:39:60:DB:47:85:9F:78:EB:A6:8E:B5:27:91:8C:AA:34:4E:73:DD:A4:1B:65:85\r\na=extmap-allow-mixed\r\na=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=setup:active\r\na=mid:0\r\na=sendrecv\r\na=sctp-port:5000\r\na=ice-ufrag:zyyLdoWNpFiIRUXn\r\na=ice-pwd:AzLMxJJrliwDFEHzpSwODjFrqlCtYsbv\r\na=candidate:1408731006 1 udp 1694498815 XX.XX.XX.XX 52584 typ srflx raddr 0.0.0.0 rport 52584\r\na=candidate:1408731006 2 udp 1694498815 XX.XX.XX.XX 52584 typ srflx raddr 0.0.0.0 rport 52584\r\na=end-of-candidates\r\n"}
2024/11/20 00:46:05 ----------------
Copy that and paste it into the client's Terminal C. If everything goes right, the WebRTC datachannel should open at both the client and the proxy.
It looks like the blocking has at least temporarily stopped since Thursday. Most snowflake connections from our vantage point have been succeeding for the last few days. The number of snowflake users from Russia seems to be climbing again.
Nobody seems to mention one thing outside of russian-speaking forums. There's the thread on NTC discussing seemingly random inaccessibility of Hetzner IPs from Russia. In post 435 one user found the culprit of immediate blocking of (all/most?) Hetzner IPs. It is enough to visit https://www.phpmyadmin.net in your browser to trigger block (no answer to TCP SYN to any Hetzner IP).
I didn't believe it at first, but decided to test. I'm an Arch Linux user, most of arch infra is hosted on Hetzner, and I've never had any problem with downloading from AUR, or reading archwiki from Russia (home, non-mobile internet). But as soon as I have visited the forementioned site, traffic to Hetzner got blocked, and I couldn't open neither AUR nor archwiki (no answer to TCP SYN). It lasted for at least 30 minutes, I had to go and turn off the computer so I didn't check for longer, but people on NTC told the block can last for up to 2 hours. The block didn't affect phpmyadmin site itself, it opened perfectly fine all that time.
phpmyadmin site is the default front of Tor Snowflake, so obviously the censor targets that site, as we can see, in interesting way. I wonder if other hoster's IP ranges are blocked too, or monitoring of user's traffic becomes stricter after that (maybe WebRTC connections succsess rate drops or something else, this requires additional study).
Accessing phpmyadmin's CNAME address (1115546720.rsc.cdn77.org) doesn't seem to trigger block.
Also in my opinion choosing phpmyadmin website as front is bad (or worse than previous domains). Unlike previous fronts like cdn.sstatic.net or ajax.aspnetcdn.net, there are less people to visit phpmyadmin site directly (installing from repos, using instructions in someone's blogs to set it up rather that reading official instruction on official website), and these people are very specific and narrow kind of people (developers), unlike regular people visiting regular sites with js cdns linked in them. But I guess Tor devs don't have many options since more and more cdn are blocking domain fronting. But you could have provided some address for tech-savvy users to set up fronting with user's own domain (domain borrowing or whatever it's called), so power user and tor broker can meet and just negotiate through some cdn that doesn't have blocking and without censor seeing tor broker domain. I wonder whether that would work, did someone propose that?
with user's own domain (domain borrowing or whatever it's called)
It's called "steal oneself," even if this is not quite correct English.
But I guess Tor devs don't have many options since more and more cdn are blocking domain fronting. But you could have provided some address for tech-savvy users to set up fronting with user's own domain (domain borrowing or whatever it's called), so power user and tor broker can meet and just negotiate through some cdn that doesn't have blocking and without censor seeing tor broker domain. I wonder whether that would work, did someone propose that?
You can do that easily just changing the "fronts=www.phpmyadmin.net" part of the bridgeline. You'll need use a domain pointing to cdn77.
If what you want is to host it in a different CDN that is a bit more of work, and you'll need to need to configure it to forward the traffic to the broker. We don't have a guide for that currently. But you can just use apmcache or sqs instead of domain fronting: https://forum.torproject.org/t/new-sqs-rendezvous-method-for-snowflake/11713
But I have the feeling that the hetzner block is not related to snowflake, as not so many snowflake proxies are in hetzner, but it might be an attempt to block bridges. As many bridges are hosted in hetzner and we use the same domain front for requesting bridges from Tor Browser.
I had to go and turn off the computer so I didn't check for longer, but people on NTC told the block can last for up to 2 hours
From my experience the block is indefinite, I had it for 2 days until I disconnected my internet for 15 minutes. I assume it would not go away at all without that. That said I also have a static ip address which might be the reason.
I was reading different English speaking places that discuss that block and I'm surprised that there's not a lot of mentions of OVH ips being blocked as well, which happens together with Hetzner, which is the case for me and a lot of people on NTC.
Tor 0.4.9.2-alpha (git-b38ce105086e06aa).
Snowflake bridges
snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://1098762253.rsc.cdn77.org/ fronts=www.cdn77.com,www.phpmyadmin.net ice=stun:stun.antisip.com:3478,stun:stun.epygi.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.mixvoip.com:3478,stun:stun.nextcloud.com:3478,stun:stun.bethesda.net:3478,stun:stun.nextcloud.com:443 utls-imitate=hellorandomizedalpn
snowflake 192.0.2.4:80 8838024498816A039FCBBAB14E6F40A0843051FA fingerprint=8838024498816A039FCBBAB14E6F40A0843051FA url=https://1098762253.rsc.cdn77.org/ fronts=www.cdn77.com,www.phpmyadmin.net ice=stun:stun.antisip.com:3478,stun:stun.epygi.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.mixvoip.com:3478,stun:stun.nextcloud.com:3478,stun:stun.bethesda.net:3478,stun:stun.nextcloud.com:443 utls-imitate=hellorandomizedalpn
snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=foursquare.com ice=stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 utls-imitate=hellorandomizedalpn
snowflake 192.0.2.4:80 8838024498816A039FCBBAB14E6F40A0843051FA fingerprint=8838024498816A039FCBBAB14E6F40A0843051FA url=https://snowflake-broker.torproject.net.global.prod.fastly.net/ front=foursquare.com ice=stun:stun.l.google.com:19302,stun:stun.antisip.com:3478,stun:stun.bluesip.net:3478,stun:stun.dus.net:3478,stun:stun.epygi.com:3478,stun:stun.sonetel.net:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.voys.nl:3478 utls-imitate=hellorandomizedalpn
Connection status: 10%
That's all, it doesn't work, alas.
No luck with that and those as well.
@sergeevabc, please try this method too:
snowflake 192.0.2.5:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net/ ampcache=https://cdn.ampproject.org/ front=www.google.com ice=stun:stun.antisip.com:3478,stun:stun.epygi.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.mixvoip.com:3478,stun:stun.nextcloud.com:3478,stun:stun.bethesda.net:3478,stun:stun.nextcloud.com:443 utls-imitate=hellorandomizedalpn
snowflake 192.0.2.6:80 8838024498816A039FCBBAB14E6F40A0843051FA fingerprint=8838024498816A039FCBBAB14E6F40A0843051FA url=https://snowflake-broker.torproject.net/ ampcache=https://cdn.ampproject.org/ front=www.google.com ice=stun:stun.antisip.com:3478,stun:stun.epygi.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.mixvoip.com:3478,stun:stun.nextcloud.com:3478,stun:stun.bethesda.net:3478,stun:stun.nextcloud.com:443 utls-imitate=hellorandomizedalpn
@gusgustavo No luck with them, too. On the bright side, obfs4, conjure and webtunnel work.
Connection status: 10%
Thanks for letting us know. It's difficult for us to tell how snowflake is failing or being blocked without more detailed logs. Can you open a confidential issue with your tor logs?
I use stun to occasionally connect to my machine behind full-cone NAT, and I've noticed that stun protocol was blocked from mid-summer till today (it worked in June last time when I've used public stun to punch through NAT). Seems that only stun to public stun servers like stun.l.google.com, stun.bethesda.net were affected. Stun between webtunnel client and other webtunnel peer is not affected as can be seen in attached captures below. The only stun server that was accessible all that time was stun.rtc.yandex.net (of all the gists and lists of public stun servers I could find).
I've decided to test whether snowflake works in these conditions, and it does, even though stun.l.google.com is blocked. Had to change front though:
Bridge snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://1098762253.rsc.cdn77.org/ fronts=<...> ice=stun:stun.l.google.com:19302 utls-imitate=hellorandomizedalpn
Bridge snowflake 192.0.2.4:80 8838024498816A039FCBBAB14E6F40A0843051FA fingerprint=8838024498816A039FCBBAB14E6F40A0843051FA url=https://1098762253.rsc.cdn77.org/ fronts=<...> ice=stun:stun.l.google.com:19302 utls-imitate=hellorandomizedalpn
where <...> is another website using cdn77 I know. www.cdn77.com is blocked by SNI (session freezes after client hello), and www.phpmyadmin.net, though it doesn't cause hetzner to be blocked anymore, might cause stricter filtering or other side-effects.
If you don't know, our government is currently fighting voice-over-ip services, throttling and blocking WhatsApp and Telegram voice calls, blatantly lying that they're fighting fraud calls, and blocking every possible voip technology these "fraudsters" move into next (threatened Google Meet, Zoom), so blocking stun servers kinda fits this logic in this time period (it could break functionality of other voip services like Jitsi, but they care about collatreal damage less and less), moreso it could've helped to block snowflake and boost mobile ISP's profits damaged by voip-services (lobbyism), mobile shutdowns and website white-lists (yes we have white lists on mobile internet starting this summer, I don't use mobile internet but people around me are obsessed with looking for right SNI, sticking it into xray reality pointing to vps of vk/yandex cloud, daisy-chaining that vps to another vps and bypassing white list). stun.l.google.com became accessible today however, have no idea whether temporarily or not.
Dumps and logs: https://0x0.st/KAzh.7z.gpg (encrypted with @cohosh 's key). 5th dump is today, with google stun server accessible, other 4 are from 2 days ago.
Upd: turned out the block is inconsistent and random, one stun request to stun.l.google.com can result in response if you keep sending them one after another, and sometimes it's 100% response rate. Seems to be dependant on path ip packets are sent by ISP.
@kad09 thanks for letting us know and for the update. We've been finding our CDN77 domain fronting settings to be inconsistently blocked in other places as well. We discussed it our team meeting today and have some plans to ship changes to defaults and recommended settings in Tor Browser and other applications.
In the meantime, you might have more success with ampcache for rendezvous instead of domain fronting:
Bridge snowflake 192.0.2.5:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://snowflake-broker.torproject.net/ ampcache=https://cdn.ampproject.org/ front=www.google.com ice=stun:stun.antisip.com:3478,stun:stun.epygi.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.mixvoip.com:3478,stun:stun.nextcloud.com:3478,stun:stun.bethesda.net:3478,stun:stun.nextcloud.com:443 utls-imitate=hellorandomizedalpn
You can also try these netlify domain fronted bridges:
Bridge snowflake 192.0.2.3:80 2B280B23E1107BB62ABFC40DDCC8824814F80A72 fingerprint=2B280B23E1107BB62ABFC40DDCC8824814F80A72 url=https://voluble-torrone-fc39bf.netlify.app/ fronts=vuejs.org ice=stun:stun.antisip.com:3478,stun:stun.epygi.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.mixvoip.com:3478,stun:stun.nextcloud.com:3478,stun:stun.bethesda.net:3478,stun:stun.nextcloud.com:443 utls-imitate=hellorandomizedalpn
Bridge snowflake 192.0.2.4:80 8838024498816A039FCBBAB14E6F40A0843051FA fingerprint=8838024498816A039FCBBAB14E6F40A0843051FA url=https://voluble-torrone-fc39bf.netlify.app/ fronts=vuejs.org ice=stun:stun.antisip.com:3478,stun:stun.epygi.com:3478,stun:stun.uls.co.za:3478,stun:stun.voipgate.com:3478,stun:stun.mixvoip.com:3478,stun:stun.nextcloud.com:3478,stun:stun.bethesda.net:3478,stun:stun.nextcloud.com:443 utls-imitate=hellorandomizedalpn