Properly configured Shadowsocks servers reportedly blocked in China?

[gfw-report]

trafficstars

Since March 2021, at least three users have reported that their Shadowsocks-libev servers became not pingable (temporarily in some cases) even after following the recommended setup in this tutorial (中文版). The tutorial documented how to setup a Shadowsocks-libev server that could defend against all known active probing attacks from the GFW as well as the partitioning oracle attack.

We document their reports in this post and we are taking immediate actions to investigate the causes.

At the same time, if you ever setup your servers by following our tutorial, we welcome you to report the status of your servers, no matter they are blocked or not. You can choose to:

1. post under this thread;
2. or post in the comment section(中文评论区);
3. or contact us privately (more contact information can be found at the footer of gfw.report).

Edited on May 5, 2021: add details from additional user report

The first report says that on the night of March 3, 2021, the server was not responding temporarily and since the morning of March 4, the server had not been pingable from at least two different network. The server remained not pingable for at least six hours until the user decided to discard the server. It appears that the user had setup the Shadowsocks server as recommended:

• Used the latest version of Shadowsocks-libev at that time: shadowsocks-libev 3.3.5-1 751 latest/edge max-c-lv -;
• Used an AEAD cipher: chacha20-ietf-poly1305;
• Used a strong password generated by openssl rand -base64 18;
• Used tcp_only mode;
• Used ufw to only allow traffic to port 22 and the Shadowsocks port 24xxx;

The second report says that the server was not responding for half an hour on March 14, 2021, and has been working properly since then.

The third report says that the server was not pingable for approximately an hour on April 20, 2021, and have been working properly since then.

We were not able to find more reports on the recent blocking of Shadowsocks. Specifically, we looked up the open and closed Github issues in the following places:

• https://github.com/shadowsocks/shadowsocks-org/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/shadowsocks-libev/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/go-shadowsocks2/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/shadowsocks-rust/issues?q=is%3Aopen+is%3Aissue

[gfw-report]

自2021年3月起，至少三名用户汇报了他们的Shadowsocks服务器在使用了我们教程建议的配置后仍然出现不能ping通的情况。按照我们教程配置的Shadowsocks服务器本应可以抵御所有已知的来自GFW的主动探测以及（还未被GFW采用的）partitioning oracle攻击。

限于数据点有限，我们不能确定是审查者使用了新的检测方法，还是这只是个别用户的网络问题引发的误会。我们将迅速调查原因。与此同时，如果您曾按照这个教程配置Shadowsocks服务器，我们希望您能提供您的服务器的情况，包括被封锁了，或是运行正常的。您可以选择：

1. 在这篇帖子下留言；
2. 或是使用这个评论区;
3. 再或是私下地联系我们（我们的联系方式在gfw.report的页脚）。

[gfw-report]

After calling for more users to report the status of their Shadowsocks servers from this post and from Twitter, one more user reported that "the Shadowsocks server configured according to the tutorial has been running stably for more than two months without any problem".

We have been encouraging users to report any blocking they encountered; however, it is also important to know their servers are working properly.

[gfw-report]

An up-to-date Outline server has been blocked in China. This server, located outside of China, has been running properly for more than two months, before it got blocked. The server possibly first got blocked around Jun 24 2021. This phenomena is interesting and possibly alarming because an up-to-date Outline server should have been able to defend against all known active probing attacks from the GFW.

We confirmed this blocking on Jun 27 2021 10 AM CST. We tested the blocking by 1) sending ICMP pings to the server (ping $ip) and 2) by making TCP connections to server ports with and without Outline listening to (nc $ip $port -v). We capture the traffic on server using tcpdump.

The result showed that only the Outline listening port was blocked, not the entire IP address. Particularly, ICMP pings from China could still receive echo reply; and SYN pings to non-Outline ports from China could still receive RST/ACK.

The blocking was single directional, meaning the censor only dropped packets from server to the client; SYN packets from China could still reach the server.

[gfw-report]

We encourage users to report the status of their Outline servers, including both blocked and unblocked status.

We were not able to find more reports on the recent blocking of Outline or other Shadowsocks implementations. Specifically, we looked up the open and closed Github issues in the following places:

Outline:

• https://github.com/Jigsaw-Code/outline-client/issues?q=is%3Aopen+is%3Aissue
• https://github.com/Jigsaw-Code/outline-server/issues?q=is%3Aopen+is%3Aissue
• https://github.com/Jigsaw-Code/outline-ss-server/issues?q=is%3Aopen+is%3Aissue

Other Shadowsocks implementations:

• https://github.com/shadowsocks/shadowsocks-org/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/shadowsocks-libev/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/go-shadowsocks2/issues?q=is%3Aopen+is%3Aissue
• https://github.com/shadowsocks/shadowsocks-rust/issues?q=is%3Aopen+is%3Aissue

The only blocking report was this one, where the server using shadowsocks-rust + xray-plugs got blocked on June 8 2021.

Have you received any blocking report recently, @fortuna ?

[gfw-report]

July 1 will be the 100th anniversary of the Chinese Communist Party. Previous work shows that the censorship will usually be strengthened and advanced around these political sensitive period of time.

While we will continue monitoring the censorship on Outline and other circumvention tools, we also encourage more users, developers and researchers to pay attention to the censorship in China this week.

[moranno]

I'm using shadowsocks with chacha20-ietf-poly1305 by https://github.com/XTLS/Xray-core. Only the listening port was blocked 40 mins ago.

[fortuna]

@gfw-report I haven't heard of private servers being blocked. I know of a few providers that did not observe any issues recently.

nthlink is a notable exception. They have always seen blocked servers, but their system is designed to deal with that via aggressive server rotation. However, as a centralized service, they are vulnerable to enumeration attacks, so it may not be Shadowsocks detection.

This is the traffic from China as reported by the Outline servers that opted to share metrics: Lines are daily averages for 1, 7 and 28-day windows.

There aren't clear drops, except maybe a bit around April 19 and June 6, 2021. Keep in mind that the metrics are not from a random sample, so they may not be representative. The numbers are biased in an unknown way.

Finally, I would like to point out that the choice of client does make a difference. The Outline client as well as the current shadowsocks-libev merge the socks header and initial user data in the same initial packet, which prevents fingerprinting based on packet size. Clients using an old version of ss-libev or other implementations may not be doing that and can expose the server.

[fortuna]

Another important thing to point out: if you are using plugins, that's not Shadowsocks anymore. I don't know how fingerprintable the different plugins are, so keep in mind that they could make you more vulnerable.

[wkrp]

I received a report about a shadowsocks-rust server being blocked in early June. The symptoms matched what was reported in https://github.com/net4people/bbs/issues/69#issuecomment-869099906: unidirectional, with only server-to-client traffic being blocked.

At https://github.com/shadowsocks/shadowsocks-rust/pull/556#issuecomment-864432692 there is the hypothesis that the replay filter itself is a feature that the GFW uses to detect Shadowsocks servers. The Shadowsocks server ignores client handshakes that reuse a previously observed salt. The replay cache is periodically reset. So an attack might look like this:

1. Observe and capture a legitimate, correctly authenticated handshake.
2. Replay the capture immediately; it should be ignored by the Shadowsocks server because it reuses a salt.
3. Wait for the replay cache to be emptied (you can do this by counting connections to the server).
4. Replay the capture again; this time it should be accepted and the Shadowsocks server should send back a non-empty response.

There's currently an experiment running to test the effect of the replay filter on detectability. shadowsocks-rust with the replay filter enabled was repeatedly blocked (different ports on the same host) in the past week. Disabling the replay filter stops the server from being blocked. This suggests that the GFW is using the behavior of the replay filter in response to active probes (in shadowsocks-rust, at least, and maybe in other implementations) as a classification feature. It may be possible to frustrate this new kind of active probing by changing the way the server remembers and reacts to replays.

[fortuna]

@wkrp Interesting finding. It would be great if someone else was able to reproduce the experiment, and include the Outline Server in the test, since it behaves a bit differently.

My understanding is that the biggest issue for fingerprinting is modified replays, where you get different behavior when you modify certain specific bytes. Replay protection is a way to protect against that, but there are other ways. For instance, the coalescing of IV, SOCKS header and initial data means that if you modify any byte in the first packet you will still get the same result (timeout). You don't need the replay filter in that case.

However, you will get different results if you modify the second packet (really the second encrypted chunk, which usually matches the packet). For example:

• Full replay: get data back, closed connection
• Replay with first packet modified: timeout
• Replay with second packet modified: some data back, then timeout (maybe closed connection).

Is that a reasonable behavior? Can the attacker use that information somehow? If it's not useful to the attacker, then we don't need the filter.

~~One important thing to keep in mind is that, if you don't have the replay filter, a single client that doesn't coalesce the initial data is enough to expose the server. The attacker can leverage the connections from that single client to fingerprint the server and be confident that the server is Shadowsocks.~~

Edit: Ignore the last part. Modifying any byte in the IV, SOCKS or initial data will still lead to a timeout these days.

[gfw-report]

Thank you for reporting on the blocking, @maker2002.

While we have no evidence that the blocking is caused by the active probing attack, our initial testing indeed suggests that the Xray implementation of Shadowsocks has active probing weaknesses. These active probing attacks have been previously proposed by Frolov et al. and found being used by the GFW (Alice et al.).

We have opened an issue here: https://github.com/XTLS/Xray-core/issues/625

It is worth noting that Outline servers have fixed this weakness for a long time and we tested and confirmed that such weakness did not exist in the latest version of Outline server. In other words, this weakness cannot be the causes of the blocking of this Outline server: https://github.com/net4people/bbs/issues/69#issuecomment-869099906

[gfw-report]

Thank you for your summary and insights, @wkrp.

We left our observations and thoughts here: https://github.com/shadowsocks/shadowsocks-rust/pull/556#issuecomment-870285895

---

https://github.com/net4people/bbs/issues/69#issuecomment-870034503

It would be great if someone else was able to reproduce the experiment, and include the Outline Server in the test, since it behaves a bit differently.

We will design and conduct experiments to validate/falsify @Mygod 's hypothesis. We will include Outline.

My understanding is that the biggest issue for fingerprinting is modified replays, where you get different behavior when you modify certain specific bytes.

That's a key insight.

[moranno]

There is another report about shadowsocks-rust with xray-plugin got blocked here: https://github.com/shadowsocks/shadowsocks-rust/issues/549#issue-914457016

And raw shadowsocks blocked in 3 days: https://github.com/shadowsocks/shadowsocks-rust/issues/549#issuecomment-869287337

I have used shadowsocks-rust + v2ray-plugin for almost 1 month. It still goes smoothly. While without any plugins, I got port blocked within 3 days.

[xiaokangwang]

I believe the best thing to do now is to reproduce the replay filter toggle experiment with a tcpdump packet capture. This can provide us with detailed intelligence about the behaviour of censorship tools. Currently, the user report based intelligence gathering is not very useful other than providing clues.

If the replay filter is proven to be the vulnerability exploited by the the GFW. The method to resolve it isn't the removal of this security feature, but to return random data of a calculated length to the attacker's client instead of shutting down the connection or returning no data when a replay is detected. An attacker without the required secret key cannot tell the difference between random data or valid server responses(when done carefully). Thus, returning random data achieve the same goal as turning off the replay filter without security complications. We should always try to reduce the compromise on the security aspect of proxy implementation.

(This post express the personal opinions of the author, instead of the affiliated V2Ray/V2Fly organization.)

[fortuna]

From https://github.com/net4people/bbs/issues/69#issuecomment-870478271:

If the replay filter is proven to be the vulnerability exploited by the the GFW. The method to resolve it isn't the removal of this security feature, but to return random data of a calculated length to the attacker's client instead of shutting down the connection or returning no data when a replay is detected.

@xiaokangwang This may open up new fingerprinting possibilities: send multiple repeated packets, look at the response size entropy. If random, then it's likely Shadowsocks, especially if varies widely. The average packet size may also give a clue. This reminds me of how ShadowsocksR gets detected (random packet sizes).

[xiaokangwang]

From #69 (comment):

If the replay filter is proven to be the vulnerability exploited by the the GFW. The method to resolve it isn't the removal of this security feature, but to return random data of a calculated length to the attacker's client instead of shutting down the connection or returning no data when a replay is detected.

@xiaokangwang This may open up new fingerprinting possibilities: send multiple repeated packets, look at the response size entropy. If random, then it's likely Shadowsocks, especially if varies widely. The average packet size may also give a clue. This reminds me of how ShadowsocksR gets detected (random packet sizes).

Thanks @fortuna . That's correct, this is an issue I initially overlooked.

There are two ways to solve this: The first one is to remember the original initial response length and replay random data of that length on replay. The second one is to maintain a database of destinations and their distribution of the lengths of the first reply. Both methods have performance issues and aren't perfect.

---

Maybe the best way to resolve these kinds of issues in the long term will be to design a new protocol(or protocols) and migrate these kinds of issues at the protocol level.

Designing a new protocol that incorporates server-side random into sessions can eliminate the need for replay checking. However, it still needs to consider how should it respond to an invalid message from a remote peer.

A little out of topic thought, the root issue here is: how should a censorship resistance tool react when it encounters an invalid message. There are a lot of different behaviours that leaks information: It can read indefinitely, read a certain length, or do not read any additional data. It can write data of a certain length to the attacker, or not. It can close the connection immediately, or wait a time out, or wait indefinitely. These behaviours should vary depending on the status of the connection and the nature of the error. Is this the first message received on the connection, or a subsequent one? Is this message invalid because of an invalid authentication code, or a replay of a previous session? A more systemic analysis could be useful in solving this issue in long run.

[fortuna]

From https://github.com/net4people/bbs/issues/69#issuecomment-870300101:

It would be great if someone else was able to reproduce the experiment, and include the Outline Server in the test, since it behaves a bit differently.

We will design and conduct experiments to validate/falsify @Mygod 's hypothesis. We will include Outline.

Thanks @gfw-report!

If you use the stock Outline Server using the Outline Manager or the install script, you will get replay protection enabled.

If you need to disable it, you will have to use outline-ss-server directly. The replay protection is disabled by default in outline-ss-server. You can increase the number of recorded salts with --replay_history.

Note that the protection against replays of server data is always on, regardless of the --replay_history flag.

[gfw-report]

An up-to-date Shadowsocks-libev server has been blocked in China. This server, located outside of China, has been running properly since Jun 24 2021, before it got blocked on July 20. The server was configured strictly following this tutorial.

We confirmed this blocking by 1) sending ICMP pings to the server (ping $ip) and 2) by making TCP connections to server ports with and without Shadowsocks-libev listening to (nc $ip $port -v). We capture the traffic on server using tcpdump.

The result showed that only the Shadowsocks-libev listening port was blocked, not the entire IP address. Particularly, ICMP pings from China could still receive echo reply.

The blocking was single directional, meaning the censor only dropped packets from server to the client; SYN packets from China could still reach the server.

//cc @madeye @Mygod @zonyitoo @riobard @database64128 @RPRX

[fortuna]

@gfw-report Did the server have replay protection enabled or disabled?

How is the experiment going?

[gfw-report]

Did the server have replay protection enabled or disabled?

Yes, the Shadowsocks-libev server has replay protection enabled.

How is the experiment going?

Starting from July 12, we ran two outline servers on two different IP addresses within the same subnet outside of China. The server were configured following your instructions. The only difference is that one server has replay protection enable, while the other has replay protection disabled.

We then make two connections to each server every 15 seconds from China. The underlying proxies traffic is HTTP and TLS. Neither of the servers has been blocked yet.

[Mygod]

You might need to crank up the number of connections. Three to four days at most should be enough to trigger from my experience.

[fortuna]

Keep in mind that the Outline server and shadowsocks-libev have slightly different behaviors. I wonder if that is making a difference in detection.

@gfw-report: Was the blocked shadowsocks-libev server receiving the same pattern of traffic?

[MatthK]

My wife just moved back to the Mainland temporarily and we couldn't communicate using Signal anymore. I first tried to install the TLS-Proxy but that didn't work. A OpenVPN connection also didn't help. Then I came across this thread and gave it a try to install the Shadowsocks server using the tutorial linked above (https://gfw.report/blog/ss_tutorial/en/).

Installation was smooth and quite flawless. Using the OpenVPN connection she managed to download the Outline client and I could send her the ss-link via WhatsApp (I specifically avoided WeChat). She could connect to my Shadowsocks server and now we can use the Signal messenger again. Even a 3-way video call worked pretty well, even though it says it's not perfect for streaming.

The client also works on her PC now and she can access her Google mails and other company stuff she needs.

Since the OpenVPN is flaky but Shadowsocks seems to work just fine. Since she's the only one to use my server, I would hope it'll fly under the radar and won't get blocked, but yeah. Let's stay optimistic.

Just for information. The server is in Hong Kong and runs on a Raspi 4 with Ubuntu 20.04.

Thank you guys for that clear tutorial and all this work. Highly appreciated to stay connected.

[MatthK]

It seems my optimism was a bit too early. The VPN now seems to be no longer working. When doing a DNS Leak Test, my wife gets various IP addresses, instead of the single one from my Unbound server on my network. Is it possible that the key(s) somehow got leaked through insecure transmission (I sent her the ss:// link via WhatsApp, and avoid to send anything confidential via WeChat). Or how else could there be a DNS leak?

Would it help to just change the password? Or password and URL? Any other tips?

[cjhenck]

Hi @MatthK - I’m the Product Manager for Outline. Can you confirm the DNS Leak Test problem was occurring with the Outline Client? If so, can you give us more details such as on which platforms, with which leak test, and whether the problem persists if she restarts the client?

I’m trying to understand whether the client is crashing on her devices and that’s the source of the test discrepancy.

[xiaokangwang]

Hi, @MatthK

When doing a DNS Leak Test, my wife gets various IP addresses, instead of the single one from my Unbound server on my network.

DNS Leak Test detects if there is a misconfiguration of the DNS setting, not the proxy itself. It is used to detect whether your DNS traffic(which includes your abridged browsing history) is sent to an unintended server. You may update your system DNS, and browser proxy settings to prevent DNS leaks. It does not show who can your proxy or who have used it. It does not show whether it has been blocked or sabotaged.

If you suspect the connection credential is leaked, you can update the password to prevent other people from connecting it with the old password. This does not prevent censorship agency from blocking your proxy with the connection credential since blocking it would only require the IP address(and optionally the port number) for an ss proxy. Your connection credential may leak during transit or after it arrived on your device. On some platform, software installed on the system or the operating system itself will scan your files and upload them to the vendor's server.

Why do you think it is no longer working? Is it because of a DNS leak, or you can no longer connect to it?

P.S.: I don't think this is supposed to be a user support ticket system. (Since we are already here you can submit more feedback to help us find the true issue. I have no intention to suppress discussion.)

[MatthK]

"I don't think this is supposed to be a user support ticket system."

OK, point taken.

[cjhenck]

P.S.: I don't think this is supposed to be a user support ticket system.

Likewise acknowledged. Apologies. For Outline questions feel free to use our tracker or Reddit r/outlinevpn.

On topic - I’m still curious about if the test results from @MatthK mean that the proxy was not working and thus we cannot rely on the previous data point that it hasn’t been blocked. (Sorry for the triple negatives.) Is there other evidence that it is working? (Visiting blocked sites, Signal working, etc.)

[database64128]

@cjhenck It's clearly working. Outline client doesn't have a routing system like many other popular censorship circumvention solutions in China. When it's connected, everything gets proxied through the remote shadowsocks server, so you either have working Internet, or nothing works.

[wkrp]

I don't have a problem with helping to diagnose user issues, as long as it's relevant to the topic. Anything specific to Outline configuration is probably better handled in an Outline forum. But like @cjhenck, I am curious the two facts that @MatthK posted, which may or may not have the same cause:

1. The Outline connection stopped working.
2. A DNS leak test showed DNS queries originating from multiple IP addresses, not the IP address of the DNS caching resolver which the Outline server is (presumably) configured to use.

If the Outline connection stopped working, there could be many causes. There could be a problem at the server, or at the client, or there could be some kind of interference in the middle (such as the server being blocked by a firewall). What specifically is going wrong is something you can troubleshoot with the Outline devs. But it would be consistent with other observations that the GFW has somehow identified the server as an Outline server and blocked its IP address and port number.

An easier way to check if requests are going through the Outline proxy (easier than doing a DNS leak test) is visit an IP-checker web site (e.g.), or just Google "my IP address", and see if it matches the IP address of the Outline server. If it does not match, but the client is able to browse various sites, then there's a good chance there's some kind of misconfiguration or the Outline client just isn't running. As @database64128 says, it's unlikely that Outline would leave you with a "half-working" setup.

I don't know how the DNS leak test works, but it's possible that even if the Outline server is configured to use a designated resolver, the test reports the IP addresses of upstream recursive resolvers. I wouldn't take this test as conclusive, unless you also have results from when the VPN was working that show a difference. But the mention of DNS leaks does bring to mind a hypothesis for how Shadowsocks servers might be discovered:

Hypothesis: The GFW takes advantage of misconfigured clients with leaky DNS to help discover circumvention servers. It identifies source IP addresses that (1) make a DNS query for a name that would normally be blocked, then (2) begin a TCP session with an unclassifiable protocol. When this happens often enough, it blocks the destination IP address / port of the TCP session.

However, this hypothesis has a major flaw, which is that a Shadowsocks client with leaky DNS is likely not to work well, because of the GFW's DNS poisoning. The client would query a DNS name, get a poisoned (useless) IP address in response, give that IP address to the Shadowsocks server through the tunnel, and the Shadowsocks server would simply fail to connect. A configuration that leaks DNS would probably be noticed by the user quickly, at least in China.

Tor emits a warning ("Your application is giving Tor only an IP address. Applications that do DNS resolves themselves may leak information.") when it gets a SOCKS request without a DNS name to resolve. A Shadowsocks client could similarly warn when it gets ATYP=1 or ATYP=4.