[SIP008] Online config

[CzBiX]

Motivation

When the server changes SS config(such as address, port, method, password), the client can auto-update config without user interaction.

Overview

1. server provides the URL of the online config file.
2. client fetch and save online config.
3. client periodically checks for config updates.

Basic file format

Basically, we follows the original JSON config format of shadowsocks. The required fields are:

{
  "server": "198.199.101.152",
  "server_port": 8388,
  "password": "u1rRWTssNv0p",
  "method": "aes-256-cfb",
  "remarks": "Example 1"
}

To support multiple servers, put those server objects into one JSON array.

Extended file format

The JSON config can be extended with any client specific field. Any unsupported filed can be just ignored.

{
  // config file struct version
  "version": 1,
  "remark": "Some server",
  "servers": [
    {
      // config id, so the client can detect it's a new server or just changed password
      "id": "1a2b3c4d",
      "remarks": "Server A",
      // SS connect config, same as shadowsocks-libev
      "server": "example.com",
      "server_port": 1234,
      "password": "example",
      "method": "chacha20-ietf-poly1305",
      "plugin": "xxx",
      "plugin_opts": "xxxxx"
    },
    // another server config
    {
      "id": "1a2b3c4e",
      "remarks": "Server B",
      "server": "example.com",
      "server_port": 4321,
      "password": "example",
      "method": "chacha20-ietf-poly1305",
      "plugin": "xxx",
      "plugin_opts": "xxxxx"
    }
  ]
}

Remark

1. The server SHOULD provide HTTPS to avoid MITM, and the client MUST NOT ignore cert error.
2. The config URL MAY include query params to identify a user as authentication.
3. The config URL MAY return HTTP redirects, the client SHOULD follow it at least once.
4. The client MAY check updates on App start/open/connected, SHOULD provide a button to force a refresh.
5. The server MAY omit some field in the config, except id, server, server_port, password, method in servers.

TODO

• [x] Improvement this spec document.
• [x] Implement in clients that have server manager feature(such as shadowsocks-windows, shadowsocks-android).

[Mygod]

Where's authentication? This proposal will get GFW-ed within days. You just don't publish your server on the Internet.

[CzBiX]

只要使用了 HTTPS 并且保证 URL 的安全，GFW 就无从得知包括 URL 在内的信息。 这就拥有了一套实现简单的防窃听及认证机制。

我提出这个提案的目的主要是为了解决朋友之间共享服务器时能便捷的更新代理信息。 就算是中心化的代理站，我也想不到这个会带来什么新的问题。

朋友之间没有必要把 URL 公布到网上，代理站需要 URL 中的信息标识用户，更加不会公布 URL。

[Mygod]

So you're using a secret in URL to reveal the secret key. The second secret is authentication. And also by doing this you have effectively introduced a handshake for the protocol.

One could work around the handshake and design a totally profitable protocol out of this:

1. Set up an HTTPS server behind CDN, possibly with per-user authentication; (one could hide such page behind a seemingly innocent blog site)
2. Use techniques similar to meek to obfuscate and do authentication with the server;
3. Server sets up a random key and listens at some random port (or even a random server if you have a lot of money) and return the key-port pair;
4. Client connects using the random key-port pair.

[madeye]

I suggest to host a QR code on a secret HTTPS website. It should do the same trick.

[CzBiX]

@Mygod I still didn't get the point about add authentication over HTTPS, and a complicated way at that. Keep URL secret should be enough. If it really needs, HTTP auth also works well. (URL like https://user:[email protected])

[Mygod]

Your URL can't be a well-known fixed string. And therefore there will be a redundant second secret.

[Mygod]

The point is that one can design a better protocol with a secret URL as a single entry point than forcing it into the current framework (i.e. as an SIP), not that per-user authentication is necessary.

[CzBiX]

@madeye So, I have to notify everyone about config changed. Users have to scan QR code many times to add servers, find out which one is the new server, which one should be removed.

Yeah, We need more manual action to live in the modern digital world.

I think my servers is not good and stable as you have. And some of my friends don't meet the tech requirements to across the wall.

Deeply hope similar feature implemented in the future.

[madeye]

@CzBiX I agree that this proposal would be useful in many scenarios. In fact, it's also easy to implement. I believe @Mygod can implement it in one line with Scala. 😄

However, to become a new feature, we need to consider more, for example, the authentication requirement here.

I think you can keep thinking about this proposal, polish your design and collect more feedback from the community. If so, I guess we would finally have a new SIP.

[madeye]

Reopen the issue to allow @CzBiX refine his proposal further.

[CzBiX]

@madeye Thanks.

Sorry, maybe I don't know much about Internet security, I still can't understand why we need more authentications. If I choose to share config via SS URI or QR code as before, it also is well-known info. What's the difference? Should we move password outside SS URI to force user input it?

In my mind, the client can use HTTPS cert to verify server's identity, and server identifies client use auth info exists in URL, the provider can choose which security level and the method they want:

• Keep URL be secret, so it works just like the password. Of course, simple URL like /ss_config.json is equivalent to a simple password, and service provider should avoid it.
• Use HTTP auth like https://user:[email protected]/config, or use token like https://example.com/config?token=xxxx.

Both of them easy to implement on server side, and easy to use because of just single URL on client side. Yes, I want to keep SS be easy to configure and use as it always does.

[Mygod]

Here's my proposed change if you really want this kind of thing.

Motivation

Current shadowsocks configuration involves many parameters. To add insult to injury, the introduction of plugins (#28) introduces even more configurations. As a result, new ss URLs are usually very bloated (#27).

All of this would be fine if the server never changes its configuration. However this is not likely to happen due to following reasons:

1. Censors are constantly getting more powerful. As a result, one may want to use different configurations to obtain the best result.
2. Even if the Shadowsocks protocol no longer changes, it's very likely that there will be changes to plugins.

Whenever the server updates its configuration, due to the handshake-less nature of the protocol, every client has to update their configuration accordingly. This can be a tedious process if the number of clients is large. Ideally we want to use only one single pre-shared key to authenticate valid users.

Protocol

Configurations via HTTPS

First, the config file is published on the Internet following one of these two URL schemes with an optional front domain parameter as the fragment: (TODO: which one?)

a. https://<domain>/<hard-to-predict-path>.ssconf[#[front-domain.example][;remark]]
b. ssconf://<domain>/<hard-to-predict-path>[#[front-domain.example][;remark]]

The web server hosting this ssconf page could be:

1. The same server as shadowsocks server or
2. A different server, or hosted on a trusted provider such as github.io, heroku, etc.

It's recommended to put the web server behind a trusted CDN, preferably the ones supporting domain fronting.

Upon GET request, the web server validates whether the request is performed using HTTPS and returns an ordinary shadowsocks json configuration in that case. Example:

{
  "server": "example.com",
  "server_port": 1234,
  "password": "example",
  "method": "chacha20-ietf-poly1305",
  "plugin": "xxx",
  "plugin_opts": "xxxxx"
}

The remark field returned from server will be ignored and the remark in URL fragment will be used instead. Multiple configurations should be put at different URLs.

Client Behavior

1. The client should take a single HTTPS URL as the entrypoint;
2. If front domain was not supplied, the client should use the domain in the URL as front domain;
3. Client resolves the supplied front domain in the DNS query and send the HTTPS GET request to the resolved address with front domain as SNI;
4. Client obtains the server's certificate;
5. Upon first connection, the client should validate the certificate against the REAL domain instead of front domain: a. If it's trusted, display a warning dialog with the server's certificate and asks the user to confirm the certificate; b. If it's not, display an error dialog with the server's certificate. Provide a hidden option (as in Chrome) for the user to confirm the certificate;
6. Otherwise, the client should validate whether the certificate has changed: a. If it changed, go to 5b; b. If not, go to 8;
7. If user confirms the new certificate, store the new certificate, otherwise go to 9;
8. Finish the HTTPS request and store the configuration in the response;
9. If there's a valid configuration in the store, use that configuration to connect, otherwise terminate.

Note that in order to guard against MitM attacks, the client now should be stateful to prevent country-scale certificate forgeries. Here's what a console ss-local would look like:

$ ss-local [--storage /path/to/protected/storage] -l 1080 --fast-open [--other-client-params] <URL>
Certificate Information for <real-domain>
  Issued to:  <real-domain>
  Issued by:  DigiCert SHA2 Extended Validation Server CA
  Other information...
This certificate looks good. Proceed to connect? [y]/n
 INFO: using tcp fast open
 INFO: initializing ciphers... chacha20-ietf-poly1305
 INFO: tcp port reuse enabled
 INFO: udprelay enabled
 INFO: udp port reuse enabled
 INFO: listening at 127.0.0.1:1080

Remark

1. URL path MUST be hard to predict (have enough entropy) but need not hide the underlying configurations. It could be something like /that-outcrop-infuriates-my-bud-my-shadowsocks-config-with-kcptun-as-plugin-japan-123.4.5.67-password-is-hello-world or /A9FEF2E0C966D101B1C5E0CCB49DCC8BE251E1E6DB53F04E11EE54C2593E6BAD36184A8684B9CE1639D7BF09FEA50CB7047536C7B252FFDA60FEC61ACE6BF57D.
2. It's not recommended to use query params and/or HTTP authentication to do authentication. Use url path instead. For example one could use /config/<secret-tag>. No matter which one you choose, the server MUST return HTTP 404 if authentication fails instead of the traditional 401 or 403.
3. It's recommended to host a seemingly innocent website for other requests, such as a blog.
4. Clients should treat redirects (HTTP 3xx) as errors. due to possible certificate issues. Web servers should handle URL changes internally.
5. Upon configuration updates, it shouldn't introduce changes that would result in IP blocks for old clients. For example, whenever password and/or method changes, the port should change as well if possible.
6. It's not recommended to use the same server as the web server without CDN. It's harder to set up, it makes the server more vulnerable and easier to fingerprint, but it shouldn't be too problematic.
7. Password in online configurations should still have as much entropy if not more since the user won't ever need to enter them manually.
8. Which URL scheme should we use? Both of them should work fine for Android. The first one looks better but I suspect only the second one can be imported with one click under browsers like Chrome.

[ghost]

I think it's better to use a separated program / script to handle this logic (like ss-manager, but for client side), and keep ss-local simple.

[Mygod]

It's just a demo of usage, and also how complicated it is to implement securely. (it's nothing you can do with one line of scala)

[WordlessEcho]

~~Deleted~~

[fortuna]

I like the idea of an online config. It can be blocked the same way the Shadowsocks server is. However, it's something you access rarely, so it will draw a lot less attention and it will be a lot harder to find and block.

I have some thoughts about practicality of the idea.

Colocation

Quite often the most convenient is to have the config live where the server is, but that means the config will be blocked if the server is. But there are solutions:

• Multiple IPs: DigitalOcean givies you a free floating IP per server, so you can have one IP for Shadowsocks and one IP for the webserver with the config. Other cloud providers have similar features, though it may not be free. This solution is convenient to those that don't have a domain name.
• Set up a CDN: You can configure Cloudflare for free as a reverse proxy for your config. Besides giving you an alternate address, it allows for domain fronting. However I believe it requires the admin to provide a domain name.

One issue we ran into in Outline is that users often don't have a domain name. Which means they can't get a TLS certificate from Let's Encrypt. Because the certificates are self-signed, we need a certificate fingerprint to validate them. The configuration URL should provide a way to specify the TLS certificate fingerprint, so clients can validate certificates when content is served from an IP address.

Publishing

Alternatively, you may publish the config somewhere else. You'll want to automate that, which means generating some API key and wiring your code properly, which is often not quite straightforward if you've never done it before.

Notice that you don't really need the fetch URL to be private. The config page can be encrypted and published publicly anywhere. It may even use HTTP, but you will want an AEAD cipher to validate authenticity. The config URL can have the secret key used to decrypt the content by the client. Ideally in the fragment, to make sure it's never sent to the host server.

Publishing requires you to trust the host, which is not true for many people, unless you use the encrypted approach above.

Some options for publishing:

• AWS or some cloud storage that is not blocked. This requires setting up cloud storage, which is complicated, and often requires payment. If you are using a secret URL, make sure to disable file listing!
• Publish to Github. This is a free solution, but requires setting up a github account and repository. You could use a Gist instead, which just requires an account. You can list the Github contents, so it requires the encrypted content approach.
• Google Drive or other drivers. Requires having a Google account and uploading the config. It's blocked in China though.
• Google Sites, Blogger. An alternative to Drive, but also blocked in China.

URL Shortener

An alternative to publishing a config is to use an editable URL Shortener. In that case it could redirect directly to the ss:// link.

As in the case of publishing, you need to trust the URL Shortener.

Backward compatibility

When you update the config, make sure the old configuration keeps running until all your clients have migrated to the new config. Otherwise you will break them.

[qiuyuzhou]

A lots of people open issues for implement of this feature in Shadowsocks-NG. Recently I consider it again.

1. Could return multiple servers in one configure file. The service provider could provide a group servers updating in one endpoint.
2. After get the new online configure file, the client can delete the old profiles which has been removed from the online file.
3. Add, update or delete by identify the profile by the host:port pair.
4. Serve in a hard-to-predict-path as @Mygod suggested.
5. Only served by https (Not http) in order to keep simple and secure.
6. @fortuna has mention an AEAD cipher approach to serve the config file by http. But it's complex to implement it. Should be easy to implement it by all client and server.
7. Only in json format.

To be simple to edit, I show the example in yaml format:

# Identify the group of servers.
# Can be used by client to track the servers from same source.
# When update, the client can use the id to deleted the old servers 
# which has been removed from the server list.
id: com.xxx.foo/bar
# Use the SIP002 format URL instead of a new spec. Keep it simple.
servers:
  - ss://[email protected]:8888#Example1
  - ss://[email protected]:8888/?plugin=obfs-local%3Bobfs%3Dhttp#Example2

[fortuna]

6. @fortuna has mention an AEAD cipher approach to serve the config file by http. But it's complex to implement it. Should be easy to implement it by all client and server.

The client could leverage the same crypto that the Shadowsocks protocol uses, so it's just a matter of reusing libraries. In Outline, we could use our ShadowsocksReader.

[fortuna]

For the config URL, we should consider not only a front domain, but also a domain to resolve and get the IP address from. For example, we can resolve cloudflare.com to get IPs for any cloudflare website.

These are the things we may need in what we give the client:

• Domain to use in the SNI and as the default for IP resolution and Host header.
• Secret HTTP path to fetch
• Optional: Domain to resolve to get the IP address (or a hard-coded IP address directly)
• Optional: Domain to use in the HTTP Host header.

If we want to support self-signed certificates for users without a domain name:

• IP address of the server
• Fingerprint of the certificate
• Secret HTTP path to fetch

[ghost]

I'd like see it become standard.

Shadowsocks-windows has another experiment few years ago: https://github.com/shadowsocks/shadowsocks-windows/tree/with_online_config

My concern:

1. If online config use ssconf://, we can register protocol handler for it. (But we haven't do it for ss:// yet)
2. On-wire format should be extensible, so other implemention such as Outline-server can send extra config they (PAC URL, DNS server, etc.) in it.
3. Self-signed cert would introduce extra complexity when implemention:

• Can we update it via online config?
• Should we validate it? How to store public key?

4. For authentication, how about TLS Client Authentication?

[madeye]

We're finalizing the online config support with an extendable JSON format. Here's an example: https://gist.github.com/madeye/8451355b1baba312dae78a6c8ca4d1a1

Generally, we don't limit any custom fields in the JSON config file. The only required fields are:

  {
    "server": "198.199.101.152",
    "server_port": 8388,
    "password": "u1rRWTssNv0p",
    "method": "aes-256-cfb",
    "remarks": "Example 1"
  }

[ghost]

More about authentication

1. POST a form(or json or other stuff) to server is traditional but still useful.
2. Custom HTTP header (Like X-Token) is a popular option.
3. Combine 1 and 2, and add some web technology, we can get this:
   1. Client POST user name and password to login URL
   2. Server allocate a token (generate by JWT?) to client
   3. Client GET online config URL with given token in custom header
   4. Server check the token and send the config

Option 1 is easy, but require POST Option 2 only need GET and also easy to implement Option 3 is hard to implement but easy to manage

[madeye]

GET is good enough, as we're forcing HTTPS URL for any online config.

You can put any restful token in your URL for authentication.

[madeye]

The latest shadowsocks-android has already supported this SIP: https://github.com/shadowsocks/shadowsocks-android/releases/tag/v5.0.3

[quekangkang]

写了一个转换 ss:// 地址为可用于订阅的 json 库，可以参考： https://github.com/quekangkang/ss2json

[fortuna]

@madeye what's the URL scheme you are using in ss-android?

@studentmain did you release this for ss-windows?

[ghost]

@madeye what's the URL scheme you are using in ss-android?

@studentmain did you release this for ss-windows?

We have plan to do that, but heavy refactor is expected...

[ghost]

@studentmain did you release this for ss-windows?

Now I'm preparing for it. Maybe few versions later. We will use HTTP scheme (http:// and https://) for online config. BTW, ss-windows now registered ss:// URL handler on Windows system when user requested.

[database64128]

We have released shadowsocks-windows v4.2.1.0, which contains support for SIP008 online configuration.

[database64128]

TL;DR: it would be much easier for clients to implement SIP008, if we only have one JSON format.

The original JSON format is extensible, flexible, and can be easily supported by clients. The basic file format added by @madeye, however, seems to exclusively favor shadowsocks-android over the neutral stance a standard should take. It also significantly and unnecessarily complicates the parsing process, which is already causing compatibility issues. Top-level JSON arrays are also known to have potential security vulnerabilities.

I propose we revert to the version of format drafted by the original author, currently listed as the extended file format as edited by @madeye, to become a simple yet highly extensible protocol standard, to minimize compatibility risks, to make it easier for servers and clients to implement.

Some typos in the original version are fixed in the following version:

{
    "version": 1,
    "servers": [
        {
            // Server UUID to distinguish between servers when updating.
            "id": "27b8a625-4f4b-4428-9f0f-8a2317db7c79",
            "remarks": "Name of the server",
            "server": "example.com",
            "server_port": 8388,
            "password": "example",
            "method": "chacha20-ietf-poly1305",
            "plugin": "xxx",
            "plugin_opts": "xxxxx"
        },
        // Another server
        {
            "id": "7842c068-c667-41f2-8f7d-04feece3cb67",
            "remarks": "Name of the server",
            "server": "example.com",
            "server_port": 8388,
            "password": "example",
            "method": "chacha20-ietf-poly1305",
            "plugin": "xxx",
            "plugin_opts": "xxxxx"
        }
    ]
    // The above fields are mandatory.
    // You may add custom fields.
}

Maybe we don't need the ssconf:// protocol.

I have mixed feelings about the proposed ssconf:// protocol. I understand that some client implementations would like to register a protocol handler for it. But personally I prefer it if we stick with the underlying protocol. It is no different than an ordinary http(s):// link, after all.

@fortuna @alalamav I noticed Jigsaw-Code/outline-shadowsocksconfig added support for ssconf://. We'd like to hear your opinions on it.

Development plans of some clients

Currently shadowsocks-windows has support for both JSON formats. Qv2ray only supports the extended file format. Both clients have no support for ssconf://.

As we work on shadowsocks-windows v5, the SIP008 support is likely to be reimplemented. And I'm seriously considering dropping support for the basic file format. I have discussed it with Qv2ray developers and some V2Fly members. They also believe the extended file format is the way to go.