[RFC] Flags for customizing service creation and links

[josegonzalez]

Rather than have a bunch of semi-related issues open, I'll open a single issue to track the various flags we want to add:

Creation Flags

• dokku/dokku-mongo#37
• dokku/dokku-postgres#61
• dokku/dokku-elasticsearch#28

We currently allow some values to be configured via environment variables. This doesn't work as well when users are running commands over ssh. We should instead allow users to specify any of these initial values during the creation step.

• [x] service:create: Add the ability to specify a flag --config-options "OPTS" to define any extra options.
  • Mongo specific
• [x] service:create: Add the ability to specify a flag --custom-env "USER=alpha;HOST=beta" to define the custom env.
• [x] service:create: Add the ability to specify a flag --image IMAGE to define the image name.
• [x] service:create: Add the ability to specify a flag --image-version IMAGE_VERSION to define the image version.
• [x] service:create: Add the ability to specify a flag --password PASSWORD to define the password.
  • Should include a warning that doing this may not be as secure as our random password.
• [x] service:create: Add the ability to specify a flag --root-password PASSWORD to define the root password.
  • Should include a warning that doing this may not be as secure as our random password.

Clone Flags

• dokku/dokku-redis#105

Upgrading/Downgrading is now impossible since https://github.com/dokku/dokku-postgres/commit/6a86efcd5de31613fb88544eedc0c3c716b98d38, and we should allow folks to easily override the environment variables via flags.

• [x] service:clone: Add the ability to specify a flag --image IMAGE to define the image name.
• [x] service:clone: Add the ability to specify a flag --image-version IMAGE_VERSION to define the image version.

Link Aliases

• dokku/dokku-redis#52
• dokku/dokku-mongo#44

We have a finite number of services that can be linked at once to an application, limited by a hardcoded list of colors. When that list is depleted, we'll have an infinite loop where we cannot link a service to an app. Users may also wish to specify the exact alias being used for the link so that it is easy to predict what the url for a service will be.

• [x] service:link: Fail hard when the max number of automatic alias names is reached.
• [x] service:link: Add the ability to specify a flag --alias ALIAS for the purposes of defining the exact link alias.
  • The link value should not be prefixed or suffixed with either the name of the plugin or _URL.

Memcached Memory Limit

• dokku/dokku-memcached#21

Memcached is currently limited to 64 megabytes. This is probably good for most of our users - people use Dokku on memory-limited servers - but there are cases where they may wish to use more memory.

• [x] service:create: Add the ability to specify a flag --memory MB for the purposes of defining the memory in use.
  • Default to 64, and write this value to disk. This should be used anytime the container is recreated.

Export Options

• dokku/dokku-mariadb#56

• [ ] service:export: Add the ability to specify a flag --export-options "OPTS" to define any extra options.

  • Mariadb/MySQL specific
  • Ignore for the backup automation commands for now.

Querystring Values

• dokku/dokku-postgres#56

Some frameworks use querystring values as a way to specify conditions for the connection. For instance, you might set the connection collation or pool size.

• [x] service:link: Add the ability to specify a flag --querystring "key=value&key=value".
  • Validate that the querystring does not begin with a ?. Chop it off it is is there.
  • Do not perform any other validation on this value. If people want to screw it up, not much we can do.
  • Just append it to the end of the url if it is non-zero in length, after a ?.

Single Container Usage

• dokku/dokku-mariadb#35
• dokku/dokku-mysql#44
• dokku/dokku-postgres#49

Users on smaller VPSs will want to save memory by not running many containers for a database service. This will allow us to mark all community service plugins as deprecated.

• [ ] service:create-database <service> <database-name>: Create this command.
  • Store the list of databases in /var/lib/dokku/services/postgres/SERVICE/databases/NAME, where each file is simply the name of a database.
• [ ] service:create: Store the user/pass info on disk in a new format:
  • Store the user/pass info in a set of files located in /var/lib/dokku/services/postgres/SERVICE/auth/USERNAME, where the username is the name of the user, and the contents are the password. Note that we will need to migrate all the old auth stuff over, including any root users.
  • Users will have access to everything in the service. If a developer needs to lockdown a specific user, that's on them to coordinate later.
• [ ] service:create-user <service> <username> <password>: Create this command.
• [ ] service:link: Add the ability to specify a flag --user USERNAME to specify the user that will be set for a specific link.
• [ ] service:link: Add the ability to specify a flag --database DATABASE to specify the database that will be set for a specific link.

[josegonzalez]

Useful stackoverflow comment on argparsing. Repost follows:

---

As other people explained, getopts doesn't parse long options. You can use getopt, but it's not portable (and it is broken on some platform...)

As a workaround, you can implement a shell loop. Here an example that transforms long options to short ones before using the standard getopts command (it's simpler in my opinion):

# Transform long options to short ones
for arg in "$@"; do
  shift
  case "$arg" in
    "--help") set -- "$@" "-h" ;;
    "--rest") set -- "$@" "-r" ;;
    "--ws")   set -- "$@" "-w" ;;
    *)        set -- "$@" "$arg"
  esac
done

# Default behavior
rest=false; ws=false

# Parse short options
OPTIND=1
while getopts "hrw" opt
do
  case "$opt" in
    "h") print_usage; exit 0 ;;
    "r") rest=true ;;
    "w") ws=true ;;
    "?") print_usage >&2; exit 1 ;;
  esac
done
shift $(expr $OPTIND - 1) # remove options from positional parameters

[MichaelSp]

Is there an ETA for the "Single Container"-Feature?

[josegonzalez]

Whenever I get to it. Dokku doesn't pay the bills, so if you want to help change that, let me know and I can get you an ETA.

[javierav]

@josegonzalez I am interested in the "Single Container" feature, how can I help with this issue?

[lostb1t]

@josegonzalez Im working on the single container feature for postgres. Wondering if it would be easier and more secure to just drop the --user flag? just let dokku handle it, every db has it's own user/pass. More secure then having one user for multiple database's, and it keeps the plugin syntax more simple. In my opinion the additonal value the --user flag is neglectable

[anxolerd]

@josegonzalez I'd like to implement single-container feature for mariadb plugin. Though I feel I don't understand RFC for this completely. For example, currently I have troubles understanding the following:

1. Should there be only one container per database version? Or both workflows (multi-container and single-container) have to be supported?
2. If only single-container scenario should be supported, what is the purpose of service:create?
3. What should happen when service:create-database invoked without prior service:create?

Can you please provide sample workflow with service:create and service:create-database which explains better what should every command do and how should these commands play together?

[josegonzalez]

The idea here is that the plugin would support both single and multi-container setups. It would default to multi-container, but a user could take a service and add a "database" to it (whatever that means for the underlying datastore) and then link that specific database/service pair to an application. A service must still be created in order for you to add a new database to it.

# creates the service
dokku mysql:create lollipop

# creates a new database
dokku mysql:create-database lollipop paintshop

# links the default lollipop database to your app
dokku service:link lollipop app

# links the paintshop database to your app
dokku service:link lollipop app --database paintshop

[anxolerd]

Okay. Then how do we know in which service mysql:create-database paintshop will create a database? I assumed, that create-database is the same as create but instead of creating a lot of containers, it creates databases in one special container for those purposes.

Also I didn't get how dokku service:link lollipop app --database paintshop should work. How paintshop database (which is created somewhere via create-database) is related to lollipop service?

Or the idea is not about having every db in one container, but about having a single container for group of related projects? For example, when I have a group of related projects about fishing, I should do the following:

# create service for group of projects
dokku mysql:create fishingprojects

# db for project one
dokku mysql:create-database fishingprojects nemo

# db for project two
dokku mysql:create-database fishingprojects Dora

# links the database to your app
dokku mysql:link fishingprojects app1 --database nemo

# links the database to your app
dokku mysql:link fishingprojects app2 --database dora

And another group of projects will have their own container and databases.

[josegonzalez]

Sorry yeah I meant to add the service name to create-database.

I don't know anything about a group of projects. How users apply the single-container run mode is up to them.

Note that it should be multi container by default.

[decentral1se]

:tada: :tada: :tada:

[Alir3z4]

Could postgresql container memory limit get included in that list ?

Ref https://github.com/dokku/dokku-postgres/pull/188