Docker secrets

[mentalsoup]

What feature would you like added?

When running orbital-sync on a docker swarm there is the option to use secrets in place of sensitive data for environment variables within the docker-compose.yml. An example of such a file as follows

version: "3.9"

networks: mynetwork: external: true

services: orbital-sync: image: mattwebbio/orbital-sync:latest networks: - mynetwork secrets: - pihole1_password - pihole2_password environment: - PRIMARY_HOST_BASE_URL=https://pihole1.mydomain.com - PRIMARY_HOST_PASSWORD_FILE=/run/secrets/pihole1_password - SECONDARY_HOST_1_BASE_URL=https://pihole2.mydomain.com - SECONDARY_HOST_1_PASSWORD_FILE=/run/secrets/pihole2_password - INTERVAL_MINUTES=30 deploy: restart_policy: condition: on-failure mode: global placement: constraints: [node.role == manager]

secrets: pihole1_password: external: true pihole2_password: external: true

When not in a swarm (standalone) secrets can also be declared within the compose file as an alternative:

secrets: pihole1_password: file: $SECRETS_DIR/pihole1_password

These are saved to the file system as straight text files with no extension defined.

I would caveat this with the precedent that I run mine in a home lab with no external access. I am sure there are additional steps others may wish to take which would make this more secure, but given I would never expose my DSN resolvers to the world, then I feel my efforts are sufficient for the purpose.

Version

No response

Log output

No response

[mattwebbio]

Thanks for both of these! 🙂

I'll have to look into this further as I don't have any experience with Docker secrets. https://github.com/mattwebbio/orbital-sync/issues/26 is lower hanging fruit so I may do that first.

[cbundy]

I will look at this if it's still considered since I would also like this addition

[mattwebbio]

@cbundy Would absolutely accept a PR!

[cbundy]

Here's my initial thoughts (I've not done this before).

Docker Swarm makes secrets available as files in /run/secrets/ e.g. the contents of /run/secrets/PRIMARY_HOST_PASSWORD would be your password.

The directory can be modified (secrets can be placed anywhere the user wants in the container, I don't know the use cases for this however, all mine are in the default location).

Thinking that it will be simple to create a new type of getRequiredEnv that is accepting of a secret file - if the environment variable looks like an absolute path (starts with /) then I will assume that it's a file reference and not the secret in plaintext in the ENV.

It can then attempt to read the secret, using something like this library or just taking this approach if it's not worth adding the dependency.

If it can't read the file, it will fall back to using the raw value in the ENV as a somewhat messy fallback, but I am going out on a limb to suggest not many pihole passwords start with / (watch me be wrong).

Decent approach? I'm not practiced at all at dealing with standard Linux conventions, but have some experience with Swarm.

Edit: Forgot to mention, simpler alternative may be to just support the default location of /run/secrets/ and leave it at that.

[saua]

Just a heads up that a fairly common convention is to support the environment variables with a _FILE suffix to indicate reading the content from a file (i.e. PRIMARY_HOST_PASSWORD would contain the actual value OR PRIMARY_HOST_PASSWORD_FILE would contain a file path that is read to get the value).

This way you'd avoid having to guess if the content is a valid file name (which could be problematic, especially for passwords, or configuration values that are already file paths).

[cbundy]

I have seen this on at least one other project I use to be fair, I feel like it is hit and miss between interpreting as a secret file, and requiring explicit _FILE suffix - my thought was that for a fairly niche feature, just having a go at the secret file would make it less noisy/messy for everyone else. But good to hear thoughts on it.

[mattwebbio]

On second thought, last week I took a stab at generifying the configuration implementation (https://github.com/mattwebbio/orbital-sync/pull/244) to allow pulling configuration values from other sources (i.e. a config.json). Locally, I have what I believe to be a much nicer implementation that I haven't quite ironed out yet. I can take a stab at it again tonight and see if I can get it working.

To check my understanding: to support Docker Secrets properly, all we'd have to do is accept a *_FILE env var for any configuration value and then read from that file inside the container to retrieve the value?

[cbundy]

Yes, Swarm will place the secret inside a file with the secret name (configured by the user at secret creation time) into the desired location.

Example from Docker docs

docker service create \
     --name mysql \
     ...
     --secret source=mysql_root_password,target=mysql_root_password \
     --secret source=mysql_password,target=mysql_password \
     -e MYSQL_ROOT_PASSWORD_FILE="/run/secrets/mysql_root_password" \
     -e MYSQL_PASSWORD_FILE="/run/secrets/mysql_password" \
     ...
     mysql:latest

This uses the _File convention that @saua mentioned.

[mattwebbio]

Okay, I'll try to revisit this evening and see if I can get something working - thanks @cbundy 😄

[cbundy]

Now I need to find something else to work on...

Best of luck

[mattwebbio]

TL;DR: pre-release with Docker secrets support at ghcr.io/mattwebbio/orbital-sync:pr-271 - please test!

Hey everyone! So sorry for the delay on this. You shouldn't have noticed (if I did the work right), but the Orbital Sync configuration went through a major refactor that makes it easier to support new config sources like this one!

The old environment variables formats are supported, but the preferred format for some of them changed slightly; for example SECONDARY_HOST_1_PASSWORD is now SECONDARY_HOSTS_1_PASSWORD. See the docs for the updated formats: https://orbitalsync.com/CONFIG.html

Using these changes, I now have a PR open to add Docker secrets support using the method outlined above. In other words, using the new env var formats one can set the SECONDARY_HOSTS_1_PASSWORD_FILE env var to a file containing a value and it will be read for its value. This _FILE suffix should work for all configuration options, not just passwords, though I don't see any particular reason why that would be valuable for anything other than passwords.

I have a pre-release up on GHCR if anyone is available to test; the image tag is: ghcr.io/mattwebbio/orbital-sync:pr-271. If anyone is available to test it, I'd love to get some eyes on it before I release it!

[mattwebbio]

I've also updated the included documentation: https://github.com/mattwebbio/orbital-sync/blob/f394b635e66bb970965d714fb2e83902b3476463/CONFIG.md#docker-secrets

And here is the updated example, borrowed from OP's message:

services:
  orbital-sync:
    image: mattwebbio/orbital-sync:latest
    secrets:
      - pihole1_password
      - pihole2_password
    environment:
      - PRIMARY_HOST_BASE_URL=https://pihole1.mydomain.com
      - PRIMARY_HOST_PASSWORD_FILE=/run/secrets/pihole1_password
      - SECONDARY_HOSTS_1_BASE_URL=https://pihole2.mydomain.com
      - SECONDARY_HOSTS_1_PASSWORD_FILE=/run/secrets/pihole2_password
secrets:
  pihole1_password:
    external: true
  pihole2_password:
    external: true

[jeremyhayes]

👋 Thanks for adding this!

I'm testing the canary build, but it seems like something still thinks the original password config option is required. I know the secret is good, I'm using it to deploy pihole itself.

Here's the log

file:///usr/src/app/dist/config/parse.js:67
                throw new MissingRequiredPropertiesError(path, missingRequired);
                      ^

MissingRequiredPropertiesError: Missing required properties for "primaryHost": password
    at parseSchema (file:///usr/src/app/dist/config/parse.js:67:23)
    at file:///usr/src/app/dist/config/parse.js:60:17
    at Array.map (<anonymous>)
    at parseSchema (file:///usr/src/app/dist/config/parse.js:57:62)
    at Config (file:///usr/src/app/dist/config/index.js:5:12)
    at file:///usr/src/app/dist/index.js:7:16
    at ModuleJob.run (node:internal/modules/esm/module_job:195:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:336:24)
    at async loadESM (node:internal/process/esm_loader:34:7)
    at async handleMainPromise (node:internal/modules/run_main:106:12)

Node.js v18.19.1

And here's the relevant section of my swarm's docker-compose.yml

# ...

  orbital-sync:
    #image: mattwebbio/orbital-sync:1.6.0
    image: ghcr.io/mattwebbio/orbital-sync:pr-271 # testing docker secrets
    environment:
      - PRIMARY_HOST_BASE_URL=https://pihole1.[redacted].com
      - PRIMARY_HOST_PASSWORD_FILE=/run/secrets/pihole-web-password
      - SECONDARY_HOSTS_1_BASE_URL=https://pihole2.[redacted].com
      - SECONDARY_HOSTS_1_PASSWORD_FILE=/run/secrets/pihole-web-password
      - SECONDARY_HOSTS_2_BASE_URL=https://pihole3.[redacted].com
      - SECONDARY_HOSTS_2_PASSWORD_FILE=/run/secrets/pihole-web-password
      - TZ=America/New_York
    secrets:
      - pihole-web-password

# ...

[mattwebbio]

@jeremyhayes thank you so much for taking the time to test, and so sorry for wasting your time! Turns out the pre-release flow I had set up wasn't working correctly - I've figured out the issue and fixed it. If you docker pull ghcr.io/mattwebbio/orbital-sync:pr-271 and try again, it should be working now!

[jeremyhayes]

It's alive! 🎉

3/23/2024, 7:45:41 PM: ➡️ Signing in to https://pihole1.[redacted].com/admin...
3/23/2024, 7:45:42 PM: ✔️ Successfully signed in to https://pihole1.[redacted].com/admin!
3/23/2024, 7:45:42 PM: ➡️ Downloading backup from https://pihole1.[redacted].com/admin...
3/23/2024, 7:45:42 PM: ✔️ Backup from https://pihole1.[redacted].com/admin completed!
3/23/2024, 7:45:42 PM: ➡️ Signing in to https://pihole2.[redacted].com/admin...
3/23/2024, 7:45:42 PM: ➡️ Signing in to https://pihole3.[redacted].com/admin...
3/23/2024, 7:45:43 PM: ✔️ Successfully signed in to https://pihole2.[redacted].com/admin!
3/23/2024, 7:45:43 PM: ➡️ Uploading backup to https://pihole2.[redacted].com/admin...
3/23/2024, 7:45:43 PM: ✔️ Successfully signed in to https://pihole3.[redacted].com/admin!
3/23/2024, 7:45:43 PM: ➡️ Uploading backup to https://pihole3.[redacted].com/admin...
3/23/2024, 7:45:43 PM: ✔️ Backup uploaded to https://pihole2.[redacted].com/admin!
3/23/2024, 7:45:43 PM: ➡️ Updating gravity on https://pihole2.[redacted].com/admin...
3/23/2024, 7:45:44 PM: ✔️ Backup uploaded to https://pihole3.[redacted].com/admin!
3/23/2024, 7:45:44 PM: ➡️ Updating gravity on https://pihole3.[redacted].com/admin...
3/23/2024, 7:45:48 PM: ✔️ Gravity updated on https://pihole2.[redacted].com/admin!
3/23/2024, 7:45:48 PM: ✔️ Gravity updated on https://pihole3.[redacted].com/admin!
3/23/2024, 7:45:48 PM: ✔️ Success: 2/2 hosts synced.
3/23/2024, 7:45:48 PM: Waiting 60 minutes...

[mattwebbio]

Released in v1.7!