mix_systemd

This library generates a systemd unit file to manage an Elixir release. It supports releases generated by Elixir 1.9+ mix release or Distillery.

At its heart, it's a mix task which reads information about the project from mix.exs and config/config.exs then generates systemd unit files using Eex templates. The goal is that the project defaults will generate a good systemd unit file, and standard options support more specialized use cases.

It uses standard systemd functions and conventions to make your app a more "native" OS citizen, and takes advantage of systemd features to improve security and reliability. While it can be used standalone, more advanced use cases use scripts from e.g. mix_deploy.

This complete example app puts the pieces together.

Installation

Add mix_systemd to the list of dependencies in mix.exs:

def deps do
  [
    {:mix_systemd, "~> 0.7"},
  ]
end

Configuration

The library tries to choose reasonable defaults, so you may not need to configure anything. It reads the app name from mix.exs and calculates default values for its configuration parameters. For example, if your app is named foo_bar, it will create a service named foo-bar, deployed to /srv/foo-bar, running under the user foo-bar.

You can override these parameters using settings in config/config.exs, e.g.:

config :mix_systemd,
  app_user: "app",
  app_group: "app",
  base_dir: "/opt",
  env_vars: [
    "PORT=8080",
  ]

Configuration strategies

There are four different kinds of things that we may want to configure:

1. Static information about application layout, e.g. file paths. This is the same for all machines in an environment, e.g. staging or prod.

2. Information specific to the environment, e.g. the hostname of the db server.

3. Secrets such as db passwords, API keys or encryption keys.

4. Dynamic information such as the IP address of the server or other other machines in the cluster.

Elixir has a couple of mechanisms for storing configuration. When you compile the release, it converts Elixir-format config files like config/config.exs into an initial application environment that is read by Application.get_env/3. That's good for simple, relatively static apps. It's not ideal to store passwords in the release file, though.

Elixir 1.9 releases support dynamic configuration at runtime. You can run the Elixir file config/runtime.exs when it boots or use the shell script rel/env.sh.eex to set environment vars. With these you can theoretically do anything. In practice, however, it can be more convenient and secure to process the config outside of the app. That's where mix_systemd and mix_deploy come in.

Beyond that, the configuration process depends on how complex things are.

Environment vars

The simplest thing is to set environment variables. Add individual vars to env_vars, and they will be set in the systemd unit file. Add files to env_files and systemd will load them before starting your app. Your application then calls System.get_env/1 in config/runtime.exs or application startup. Note that these environment vars are read at runtime, not when building your app.

env_vars: [
  # Set a variable, good for things that are not sensitive and don't change
  "PORT=8080",
],
dirs: [
  # create /etc/foo
  :configuration,
],
env_files: [
  # Read environment vars from the file /etc/foo/environment
  ["-", :configuration_dir, "/environment"],
]

/etc/foo/environment looks like:

DATABASE_URL="ecto://foo_prod:[email protected]/foo_prod"
SECRET_KEY_BASE="EOdJB1T39E5Cdeebyc8naNrOO4HBoyfdzkDy2I8Cxiq4mLvIQ/0tK12AK1ahrV4y"
HOST="www.example.com"
ASSETS_HOST="assets.example.com"
RELEASE_COOKIE="LmCMGNz04yEJ4MQc6jt3cS7QjAppYOw_bQa7NE5hPZJGqL3Yry1jUg=="

config/runtime.exs does something like the following (the default files are good):

config :foo, Foo.Repo,
  url: System.get_env("DATABASE_URL")

config :foo, FooWeb.Endpoint,
  http: [:inet6, port: System.get_env("PORT") || 4000],
  url: [host: System.get_env("HOST"), port: 443],
  static_url: [host: System.get_env("ASSETS_HOST"), port: 443],
  secret_key_base: System.get_env("SECRET_KEY_BASE"),
  cache_static_manifest: "priv/static/cache_manifest.json"

Copying files

The question is how to get the environment files onto the server. For simple server deployments, we can copy the config to the server when doing the initial setup.

In cloud environments, we may run from a read-only image, e.g. an Amazon AMI, which gets configured at start up based on the environment by copying the config from an S3 bucket, e.g.:

umask 077
aws s3 sync --exact-timestamps --no-progress "s3://${CONFIG_BUCKET}/" "/etc/foo/"
chown -R $DEPLOY_USER:$APP_GROUP /etc/foo
find /etc/foo -type f -exec chmod 640 {} \;
find /etc/foo -type d -exec chmod 750 {} \;

The following example runs the script /srv/foo/bin/deploy-sync-config-s3 from mix_deploy. It uses an environment file in /srv/foo/etc/environment to bootstrap the sync, e.g. setting the S3 bucket name. That file is placed there by CodeDeploy at deploy time.

config :mix_systemd,
  exec_start_pre: [
    # Run before starting the app
    # The `!` means the script is run as root, not as the app user
    ["!", :deploy_dir, "/bin/deploy-sync-config-s3"]
  ],
  dirs: [
    :configuration, # /etc/foo, app configuration, e.g. db passwords
    :runtime,       # /run/foo, temp files which may be deleted between runs
  ],
  env_files: [
    ["-", :deploy_dir, "/etc/environment"], # /srv/foo/etc/environment
  ]
  env_vars: [
    # Tell release to use /run/foo for temp files
    ["RELEASE_TMP=", :runtime_dir],
  ]

Config providers

At a certain point, making everything into an environment var becomes annoying. It's verbose and vars are simple strings, so you have to encode values safely and convert them back to lists, integers or atoms.

Config providers let you load files in standard formats like TOML.

[foo."Foo.Repo"]
url = "ecto://foo_prod:[email protected]/foo_prod"
pool_size = 15

[foo."FooWeb.Endpoint"]
secret_key_base = "EOdJB1T39E5Cdeebyc8naNrOO4HBoyfdzkDy2I8Cxiq4mLvIQ/0tK12AK1ahrV4y"

The app reads these config files on startup and merges them into the app config.

defp releases do
   [
     prod: [
       include_executables_for: [:unix],
       config_providers: [
         {TomlConfigProvider, path: "/etc/foo/config.toml"}
       ],
       steps: [:assemble, :tar]
     ]
   ]
 end

The startup scripts read the initial application environment compiled into the release, parse the config file, merge the values, write it to a temp file, then start the VM. Because of that, they need a writable directory. That is configured using the RELEASE_TMP environment var, normally set to the app's runtime_dir.

dirs: [
  :configuration,
  :runtime,
],
env_vars: [
  ["RELEASE_TMP=", :runtime_dir],
],

Config servers and vaults

You can also store config params in an external configuration system and read them at runtime. An example is AWS Systems Manager Parameter Store.

Set a parameter using the AWS CLI:

aws ssm put-parameter --name '/foo/prod/db/password' --type ‘SecureString’ --value 'Sekrit!"

While it's possible to read params in config/runtime.exs, it's tedious. Better is to grab all of them at once and write them to a file, then read it in with a Config Provider like aws_ssm_provider.

aws --region us-east-1 ssm get-parameters-by-path --path "/foo/prod/" --recursive --with-decryption --query "Parameters[]" > /etc/foo/ssm.json

defp releases do
   [
     prod: [
       include_executables_for: [:unix],
       config_providers: [
         {AwsSsmProvider, path: "/etc/foo/ssm.json"}
       ],
       steps: [:assemble, :tar]
     ]
   ]
 end

Dynamic config

You can write code to do things like query the system for the primary IP address, but cloud-init already does it. You just have to read the JSON file.

The most common use for this is setting up the VM node name. In env.sh:

CLOUD_NAME=$(jq -r '.v1.cloud_name' < /run/cloud-init/instance-data.json)
if [ "$CLOUD_NAME" = "digitalocean" ]; then
    IP_ADDR=$(jq -r '.ds.meta_data.interfaces.public[0].anchor_ipv4.ip_address' < /run/cloud-init/instance-data.json)
    DEFAULT_IPV4="$IP_ADDR"
elif [ "$CLOUD_NAME" = "aws" ]; then
    IP_ADDR=$(jq -r '.ds.meta_data."local-ipv4"' < /run/cloud-init/instance-data.json)
    # IP_ADDR=$(jq -r '.ds.meta_data."public-ipv4"' < /run/cloud-init/instance-data.json)
    AWS_REGION=$(jq -r '.v1.region' < /run/cloud-init/instance-data.json)
fi
RELEASE_DISTRIBUTION="name"
RELEASE_NODE="${RELEASE_NAME}@${IP_ADDR}"

Security

An important security principle is "least privilege". If an attacker manages to compromise the app, then they can do whatever it has permissions to do, not just what you expect. Because of that, I prefer that the account that the app runs under cannot write files, and having a writable config file that is also executed is the worst case scenario.

Usage

First, use the systemd.init task to template files from the library to the rel/templates/systemd directory in your project.

mix systemd.init

Next, generate output files in the build directory under _build/#{mix_env}/systemd/lib/systemd/system.

MIX_ENV=prod mix systemd.generate

Configuration options

The following sections describe common configuration options. See lib/mix/tasks/systemd.ex for the details of more obscure options.

If you need to make changes not supported by the config options, then you can check the templates in rel/templates/systemd into source control and make your own changes. Contributions are welcome!

Basics

app_name: Elixir application name, an atom, from the app field in the mix.exs project.

module_name: Elixir camel case module name version of app_name, e.g. FooBar.

release_name: Name of release, default app_name.

ext_name: External name, used for files and directories, default app_name with underscores converted to "-", e.g. foo-bar.

service_name: Name of the systemd service, default ext_name.

release_system: :mix | :distillery, default :mix

Identifies the system used to generate the releases, Mix or Distillery.

Users

app_user: OS user account that the app runs under, default ext_name.

app_group: OS group account, default ext_name.

Directories

base_dir: Base directory for app files on target, default /srv.

deploy_dir: Directory for app files on target, default #{base_dir}/#{ext_name}.

We use the standard app directories, for modern Linux systems. App files are under /srv, configuration under /etc, transient files under /run, data under /var/lib.

Directories are named based on the app name, e.g. /etc/#{ext_name}. The dirs variable specifies which directories the app uses. By default, it doesn't set up anything. To enable them, configure the dirs param, e.g.:

dirs: [
  # :runtime,       # App runtime files which may be deleted between runs, /run/#{ext_name}
  # :configuration, # App configuration, e.g. db passwords, /etc/#{ext_name}
  # :state,         # App data or state persisted between runs, /var/lib/#{ext_name}
  # :cache,         # App cache files which can be deleted, /var/cache/#{ext_name}
  # :logs,          # App external log files, not via journald, /var/log/#{ext_name}
  # :tmp,           # App temp files, /var/tmp/#{ext_name}
],

Recent versions of systemd (since 235) will create these directories at start time based on the settings in the unit file. With earlier systemd versions, create them beforehand using installation scripts, e.g. mix_deploy.

For security, we set permissions to 750, more restrictive than the systemd defaults of 755. You can configure them with variables like configuration_directory_mode. See the defaults in lib/mix/tasks/systemd.ex.

systemd_version: Sets the systemd version on the target system, default 235. This determines which systemd features the library will enable. If you are targeting an older OS release, you may need to change it. Here are the systemd versions in common OS releases:

• CentOS 7: 219
• Ubuntu 16.04: 229
• Ubuntu 18.04: 237

Additional directories

The library uses a directory structure under deploy_dir which supports multiple releases, similar to Capistrano.

• scripts_dir: deployment scripts which e.g. start and stop the unit, default bin.
• current_dir: where the current Erlang release is unpacked or referenced by symlink, default current.
• releases_dir: where versioned releases are unpacked, default releases.
• flags_dir: dir for flag files to trigger restart, e.g. when restart_method is :systemd_flag, default flags.

When using multiple releases and symlinks, the deployment process works as follows:

1. Create a new directory for the release with a timestamp like /srv/foo/releases/20181114T072116.

2. Upload the new release tarball to the server and unpack it to the releases dir.

3. Make a symlink from /srv/#{ext_name}/current to the new release dir.

4. Restart the app.

If you are only keeping a single version, then deploy it to the directory /srv/#{ext_name}/current.

Variable expansion

The following variables support variable expansion:

expand_keys: [
  :env_files,
  :env_vars,
  :runtime_environment_service_script,
  :exec_start_pre,
  :exec_start_wrap,
  :read_write_paths,
  :read_only_paths,
  :inaccessible_paths,
]

You can specify values as a list of terms, and it will look up atoms as keys in the config. This lets you reference, e.g., the deploy dir or configuration dir without having to specify the full path, e.g. ["!", :deploy_dir, "/bin/myscript"] gets converted to "!/srv/foo/bin/myscript".

Environment vars

The library sets env vars in the unit file:

• MIX_ENV: mix_env, default Mix.env()

• LANG: env_lang, default en_US.utf8

• RUNTIME_DIR: runtime_dir, if :runtime in dirs

• CONFIGURATION_DIR: configuration_dir, if :configuration in dirs

• LOGS_DIR: logs_dir, if :logs in dirs

• CACHE_DIR: cache_dir, if :cache in dirs

• STATE_DIR: state_dir, if :state in dirs

• TMP_DIR: tmp_dir, if :tmp in dirs

You can set additional vars using env_vars, e.g.:

env_vars: [
  "PORT=8080",
]

You can also reference the value of other parameters by name, e.g.:

env_vars: [
  ["RELEASE_TMP=", :runtime_dir],
]

You can read environment vars from files with env_files, e.g.:

env_files: [
  ["-", :deploy_dir, "/etc/environment"],
  ["-", :configuration_dir, "environment"],
  ["-", :runtime_dir, "environment"],
],

The "-" at the beginning makes the file optional; the system will start without it. Later values override earlier values, so you can set defaults in the release which get overridden in the deployment or runtime environment.

Runtime dirs

The release scripts may need to write temp files and log files, e.g. when generating the application config config files. By default, they do this under the release dir, e.g. /srv/foo/current/tmp.

For security, it's better to deploy the app using a different user account from the one that the app runs under, making the source files read only. This makes it harder for an attacker to make changes to the source and then have the app run them.

In that case, we need to set an environment var which tells the release startup scripts where they can write files. For Mix releases, that is RELEASE_TMP and for Distillery it is RELEASE_MUTABLE_DIR, e.g.:

env_vars: [
  {"RELEASE_TMP=", :runtime_dir},
]

By default systemd will delete the runtime directory when restarting the app. This can be annoying when debugging startup issues. You can set runtime_directory_preserve to restart or yes (see RuntimeDirectoryPreserve).

Starting and restarting

The following variables set systemd variables:

service_type: :simple | :exec | :notify | :forking. systemd Type, default :simple.

Modern applications don't fork, they run in the foreground and rely on the supervisor to manage them as a daemon. This is done by setting service_type to :simple or :exec. Note that in simple mode, systemd doesn't actually check if the app started successfully, it just continues starting other units. If something depends on your app being up, :exec may be better.

Set service_type to :forking, and the library sets pid_file to #{runtime_directory}/#{app_name}.pid and sets the PIDFILE env var to tell the boot scripts where it is.

The Erlang VM runs pretty well in foreground mode, but traditionally runs as as a standard Unix-style daemon, so forking might be better. Systemd expects foregrounded apps to die when their pipe closes. See https://elixirforum.com/t/systemd-cant-shutdown-my-foreground-app-cleanly/14581/2

restart_method: :systemctl | :systemd_flag | :touch, default :systemctl

Set this to :systemd_flag, and the library will generate an additional unit file which watches for changes to a flag file and restarts the main unit. This allows updates to be pushed to the target machine by an unprivileged user account which does not have permissions to restart processes. Touch the file #{flags_dir}/restart.flag and systemd will restart the unit.

working_dir: Current working dir for app. systemd WorkingDirectory, default current_dir.

limit_nofile: Limit on open files, systemd LimitNOFILE, default 65535.

umask: Process umask, systemd UMask, default "0027". Note that this is octal, so it needs to be a string.

restart_sec: Time in seconds to wait between restarts, systemd RestartSec, default 100ms.

syslog_identifier: Logging name, systemd SyslogIdentifier, default service_name

ExecStartPre scripts

Scripts specified in exec_start_pre (systemd ExecStartPre]) run before the main ExecStart script runs, e.g.:

exec_start_pre: [
  ["!", :deploy_dir, "/bin/deploy-sync-config-s3"]
]

This runs the deploy-sync-config-s3 script from mix_deploy, which copies config files from an S3 bucket into /etc/foo. By default, scripts run as the same user and group as the main script. Putting ! in front makes the script run with elevated privileges, allowing it to write config to /etc/foo even if the main user account cannot for security reasons.

ExecStart wrapper script

Instead of running the main ExecStart script directly, you can run a shell script which sets up the environment, then runs the main script with exec. Set exec_start_wrap to the name of the script, e.g. deploy-runtime-environment-wrap from mix_deploy.

In Elixir 1.9+ releases you can use env.sh, but this runs earlier with elevated permissions, so this may still be useful.

Runtime environment service

You can run your own separate service to configure the runtime environment before the app runs. Set runtime_environment_service_script to a script such as deploy-runtime-environment-file from mix_deploy. This library will create a #{service_name}-runtime-environment.service unit and make it a systemd runtime dependency of the app.

Runtime dependencies

Systemd starts units in parallel when possible. To enforce ordering, set unit_after_targets to the names of systemd units that this unit depends on. For example, if this unit should run after cloud-init to get runtime network information, set:

unit_after_targets: [
  "cloud-init.target"
]

Security

paranoia: Enable systemd security options, default false.

NoNewPrivileges=yes
PrivateDevices=yes
PrivateTmp=yes
ProtectSystem=full
ProtectHome=yes
PrivateUsers=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectControlGroups=yes
MountAPIVFS=yes
                                                                                                │

chroot: Enable systemd chroot, default false. Sets systemd RootDirectory is set to current_dir. You can also set systemd ReadWritePaths=, ReadOnlyPaths=, InaccessiblePaths= with the read_write_paths, read_only_paths and inaccessible_paths vars, respectively.

Distillery

Distillery has largely been replaced by Elixir native releases. This library works fine with it, though. exec_start_pre scripts are particularly useful in the absence of env.sh.

Configure the library by setting release_system: :distillery, e.g..

config :mix_systemd,
  release_system: :distillery,
  exec_start_pre: [
    # Run script as root before starting
    ["!", :deploy_dir, "/bin/deploy-sync-config-s3"]
  ],
  dirs: [
    :configuration,
    :runtime,
  ],
  runtime_directory_preserve: "yes",
  env_vars: [
    # Use /run/foo for temp files
    ["RELEASE_MUTABLE_DIR=", :runtime_dir],
    # expand $CONFIGURATION_DIR in config files
    REPLACE_OS_VARS=true,
  ]

Set up config providers in rel/config.exs:

environment :prod do
  set config_providers: [
    {Mix.Releases.Config.Providers.Elixir, ["${CONFIGURATION_DIR}/config.exs"]}
  ]
end

This reads files in Elixir config format. Instead of including your prod.secret.exs file in prod.exs, you can copy it to the server separately, and it will be read at startup.

The TOML configuration provider works similarly:

environment :prod do
  set config_providers: [
    {Toml.Provider, [path: "${CONFIGURATION_DIR}/config.toml"]},
  ]
end

Add the TOML config provider to mix.exs:

{:toml_config_provider, "~> 0.2.0"}

You can generate a file under the release with an overlay in rel/config.exs, e.g.:

environment :prod do
  set overlays: [
    {:mkdir, "etc"},
    {:copy, "rel/etc/environment", "etc/environment"},
    # {:template, "rel/etc/environment", "etc/environment"}
  ]
end

That results in a file that would be read by:

env_files: [
  ["-", :current_dir, "/etc/environment"],
],

Contacts

I am jakemorrison on on the Elixir Slack and Discord, reachfh on Freenode #elixir-lang IRC channel. Happy to chat or help with your projects.