Fix URL Prefix Support

[neurolag]

Description

Currently, hosting the SyncStorage service under any root URL other than / like, say, /firefox-sync, causes 401 HTTP error codes caused by mismatching Message Authentication Codes (or MACs for short) as pointed out by @ethowitz here.

Changes made in this PR add a new option public_url allowing users to specify the public facing URL to the root of the syncservers services.

This public_url option is used for determining the original request uri and perform the MAC authentication properly.

Things to Note

As explained by @kyz here, the host and port for performing the MAC authentication are taken from the Forwarded or the X-Forwarded-For and X-Forwarded-Scheme etc. headers:

https://github.com/mozilla-services/syncstorage-rs/blob/8c56cae8905325345972a4abe99c12c1fc1b012c/syncserver/src/web/auth.rs#L177-L193

It might be a good idea to swap this to perform the authentication based on public_url if specified, instead. However, I did not include this in this PR and I would love to hear what other people think about this.

Testing

1. Spin up a syncserver which is hosted under a root other than /, for example: http://localhost:8080/firefox-sync:

   services:
     web:
       image: nginxproxy/nginx-proxy
       restart: unless-stopped
       ports:
         - 127.0.0.1:8080:80
       environment:
         DEFAULT_HOST: sync.example.com
       volumes:
         - /var/run/docker.sock:/tmp/docker.sock:ro
     sync-server:
       build:
         context: .
         dockerfile_inline: |
           FROM rust AS build
           ARG SYNC_STORAGE_VERSION=0.18.2
           RUN git clone https://github.com/mozilla-services/syncstorage-rs -b $${SYNC_STORAGE_VERSION} /app
           WORKDIR /app
   
           RUN \
             apt-get update \
             && apt-get install -y libpython3-dev \
             && cargo install --path ./syncserver --features mysql --locked \
             && cargo clean \
             && apt-get remove -y libpython3-dev \
             && rm -rf /var/lib/apt/lists \
             && bash -O extglob -c 'rm -rf /usr/local/cargo/!(bin)' \
             && bash -O extglob -c 'rm -rf /usr/local/cargo/bin/!(syncserver)'
   
           FROM python:3.11 AS sync
           COPY --from=build /usr/local/cargo/bin/syncserver /usr/local/bin
           COPY --from=build /app/requirements.txt .
           RUN pip install -r requirements.txt
           CMD [ "/usr/local/bin/syncserver" ]
         target: sync
       restart: unless-stopped
       environment:
         VIRTUAL_PORT: 80
         VIRTUAL_PATH: "/firefox-sync/"
         VIRTUAL_DEST: "/"
         VIRTUAL_HOST: sync.example.com
         RUST_LOG: warn
         SYNC_HUMAN_LOGS: 1
         SYNC_HOST: "0.0.0.0"
         SYNC_PORT: 80
         SYNC_MASTER_SECRET: secret
         SYNC_SYNCSTORAGE__ENABLED: "true"
         SYNC_SYNCSTORAGE__DATABASE_URL: mysql://sync:password@sync-db/SyncStorage
         SYNC_SYNCSTORAGE__ENABLE_QUOTA: 0
         SYNC_TOKENSERVER__ENABLED: "true"
         SYNC_TOKENSERVER__DATABASE_URL: mysql://token:password@token-db/TokenServer
         SYNC_TOKENSERVER__RUN_MIGRATIONS: "true"
         SYNC_TOKENSERVER__FXA_METRICS_HASH_SECRET: secret
         SYNC_TOKENSERVER__FXA_EMAIL_DOMAIN: api.accounts.firefox.com
         SYNC_TOKENSERVER__FXA_OAUTH_SERVER_URL: https://oauth.accounts.firefox.com
         SYNC_TOKENSERVER__FXA_BROWSERID_AUDIENCE: https://token.services.mozilla.com
         SYNC_TOKENSERVER__FXA_BROWSERID_ISSUER: https://api.accounts.firefox.com
         SYNC_TOKENSERVER__FXA_BROWSERID_SERVER_URL: https://verifier.accounts.firefox.com/v2
       expose:
         - 80
     sync-db:
       image: mariadb
       restart: unless-stopped
       environment:
         MARIADB_RANDOM_ROOT_PASSWORD: "yes"
         MARIADB_USER: sync
         MARIADB_PASSWORD: password
         MARIADB_DATABASE: SyncStorage
     token-db:
       build:
         context: .
         dockerfile_inline: |
           FROM rust AS build
           ARG SYNC_STORAGE_VERSION=0.18.2
           RUN git clone https://github.com/mozilla-services/syncstorage-rs -b $${SYNC_STORAGE_VERSION} /app
   
           RUN \
             cargo install diesel_cli --no-default-features --features mysql --locked \
             && bash -O extglob -c 'rm -rf /usr/local/cargo/!(bin)' \
             && bash -O extglob -c 'rm -rf /usr/local/cargo/bin/!(diesel)'
   
           FROM mariadb AS db
           RUN mkdir -p /app/tokenserver-db
           COPY --from=build /app/tokenserver-db/migrations /app/tokenserver-db/migrations
           COPY --from=build /usr/local/cargo/bin/diesel /usr/local/bin
   
           RUN { \
               echo '#!/bin/bash'; \
               echo 'diesel --database-url "mysql://$${MARIADB_USER}:$${MARIADB_PASSWORD}@localhost/$${MARIADB_DATABASE}" migration --migration-dir /app/tokenserver-db/migrations run'; \
               echo 'mariadb -u$$MARIADB_USER -p$$MARIADB_PASSWORD -D $$MARIADB_DATABASE <<EOF'; \
               echo 'INSERT INTO services (service, pattern)'; \
               echo "SELECT 'sync-1.5', '{node}/1.5/{uid}'"; \
               echo "WHERE NOT EXISTS (SELECT 1 FROM services);"; \
               echo ""; \
               echo 'INSERT INTO nodes (\`service\`, node, capacity, available, current_load, downed, backoff)'; \
               echo "SELECT LAST_INSERT_ID(), 'http://localhost:8080/firefox-sync', 1, 1, 0, 0, 0"; \
               echo "WHERE LAST_INSERT_ID() > 0;"; \
               echo "EOF"; \
             } > /docker-entrypoint-initdb.d/tokenserver-db.sh
       restart: unless-stopped
       environment:
         MARIADB_RANDOM_ROOT_PASSWORD: "yes"
         MARIADB_USER: token
         MARIADB_PASSWORD: password
         MARIADB_DATABASE: TokenServer
   

2. Try to sync your browser against http://localhost:8080/firefox-sync/1.0/sync/1.5
3. Take note that any request pointing to http://localhost:8080/firefox-sync/1.5/* fail with a 401 HTTP code

Issue(s)

Closes #1217 and closes #1649.

[kyz]

Thanks for submitting this! I hope Mozilla consider accepting it.

It might be a good idea to swap this to perform the authentication based on public_url if specified, instead. However, I did not include this in this PR and I would love to hear what other people think about this.

Personally, yes, I would like it if public_url also sets host, port and scheme if it is defined. If it's defined by config, the service should not need to guess from headers. I believe it's also the behaviour of the previous SyncServer-1.5.

[virgoparna]

This is now broken by commit 3404150. Tried to use it so that I could get syncserver-rs to work properly.

[neurolag]

I think at time of writing you can just rebase it The commits are still perfectly compatible

Gimme a sec

[virgoparna]

Built syncserver from this pull request, but sync still fails (logs from firefox, this is where it switches to localhost again):

1762537347298 Sync.Status DEBUG Status.login: success.status_ok => success.login 1762537347298 Sync.Status DEBUG Status.service: error.login.failed => success.status_ok 1762537347298 Sync.SyncAuthManager DEBUG _findCluster returning http://localhost:8000/1.5/4/

config file in server side has public_url set.

[neurolag]

To me, it looks like you're calling the sync server by http://localhost:8000/1.5/4/ instead of its public_url.

Is this, by chance, what's going wrong?

[virgoparna]

To me, it looks like you're calling the sync server by http://localhost:8000/1.5/4/ instead of its public_url.

No, prefs.js in Firefox has same value, as public_url... And it initially connects public_url according the logs... I even tried logging out firefox account and then logging back in (which managed reset tokenserver url).

Could it be, that syncserver sends it self url as a part of some response? That /4/ at the end... Configured url in firefox ends with 1.5 And public_url in syncserver config ends with ffsync And running strings against syncserver binary I get on instance of public_url And I checked out pull request to local branch with "git fetch origin pull/1655/head:local_branc_name"

1762580651257 Services.Common.RESTRequest DEBUG GET https://public_server/ffsync/1.0/sync/1.5 200 1762580651257 Services.Common.TokenServerClient DEBUG Got token response: 200 1762580651257 Services.Common.TokenServerClient DEBUG Successful token response 1762580651258 Sync.BulkKeyBundle INFO BulkKeyBundle being created for undefined 1762580651258 Sync.Status DEBUG Status.login: success.status_ok => success.login 1762580651258 Sync.Status DEBUG Status.service: error.login.failed => success.status_ok 1762580651258 Sync.SyncAuthManager DEBUG _findCluster returning http://localhost:8000/1.5/4/ 1762580651259 Sync.SyncAuthManager DEBUG Cluster value = http://localhost:8000/1.5/4/ 1762580651259 Sync.SyncAuthManager DEBUG Setting cluster to http://localhost:8000/1.5/4/

[virgoparna]

Could it be, that syncserver sends it self url as a part of some response? That /4/ at the end... Configured url in firefox ends with 1.5 And public_url in syncserver config ends with ffsync And running strings against syncserver binary I get on instance of public_url And I checked out pull request to local branch with "git fetch origin pull/1655/head:local_branc_name"

Ok. tokenserver_rs database nodes table had localhost url in node field.. Changing it to public_url removes those http://localhosty:9000 requests.... Sync still fails, but that is probably nothing to do with this patch... "One of: no meta, no meta storageVersion, or no meta syncID. Fresh start needed."

[virgoparna]

Sync still fails, but that is probably nothing to do with this patch... "One of: no meta, no meta storageVersion, or no meta syncID. Fresh start needed."

That is probably caused by https://github.com/mozilla-services/syncstorage-rs/issues/1753 current master already has fixes for mariadb support.