migrate
migrate copied to clipboard
golang-migrate
Support environment variables in migration files using go templates
This PR adds a -template command line option that, when enabled, allows you to substitute any environment variables into your migration file with go templating. i.e. If you set the LOCAL_WAREHOUSE environment variable to MY_DB and have a migration file with the following contents:
INSERT INTO {{.LOCAL_WAREHOUSE}}.INVENTORY.RECORDS ('foo') VALUES ('bar');
it will be transformed into the following before being executed:
INSERT INTO MY_DB.INVENTORY.RECORDS ('foo') VALUES ('bar');
See https://pkg.go.dev/text/template for more information on supported template formats.
As mentioned in #747, this functionality could be done with envsubt, but that requires you to build custom scripts to do so. It ended up being just as much effort to add the functionality with go template and kept our pipelines and local development workflow more simple.
Coverage decreased (-0.04%) to 57.732% when pulling 3ffb53bc84aa16b8b0a2e9119d185c75969b0b01 on ninthclowd:feature/templating into 03613f14ac4f975eb0070a23958123c5d84e6b87 on golang-migrate:master.
@dhui it would be nice to have this functionality. Is there a chance to get it into main?
This could be a very nice feature. An use case: it will allow to customize migrations files for testings pipelines that run in parallel and that requires to have different databases for each test thread to avoid conflicts.
We currently have to trick migration files with find/replace to obtain a similar feature. The template support will be very interesting.
Yes, this would be a great feature to have merged into main. I'm having to do the envsubst trick right now and I think that's rather hacky.
I made this script file to run sql-migrate with env variables present in .env file:
- This will work for mac os.
- For linux, replace
#!/bin/shwith#!/bin/bash
migrate.sh
#!/bin/sh
# Import env variables from ./.env
set -a
[ -f .env ] && . .env
echo "DB_NAME = $DB_NAME | DB_USER = $DB_USER | DB_HOST = $DB_HOST | DB_PORT = $DB_PORT"
read -p "⚡️ (Sql-Migrate) Type: <up | d | s> " command
if [[ $command == 'up' ]]
then
sql-migrate up -env="development" sslmode=disable
elif [[ $command == 'd' ]]
then
sql-migrate down -env="development" sslmode=disable
elif [[ $command == 's' ]]
then
sql-migrate status -env="development" sslmode=disable
else
read -p "Run your own command: " custom_command
sql-migrate $custom_command -env="development" sslmode=disable
fi
@dhui Do you guys have a plan to approve it or we should stick with other solutions?
You can wrap the file when using iofs to support templating
type templateFile struct {
data any
fs.File
}
func (t *templateFile) Read(p []byte) (int, error) {
_, err := t.File.Read(p)
if err != nil {
return 0, err
}
tmpl, err := template.New("").Parse(string(p))
if err != nil {
return 0, err
}
buf := new(bytes.Buffer)
err = tmpl.Execute(buf, t.data)
if err != nil {
return 0, err
}
return copy(p, buf.Bytes()), nil
}
type templateFS struct {
data any
embed.FS
}
func (t *templateFS) Open(name string) (fs.File, error) {
file, err := t.FS.Open(name)
if err != nil {
return nil, err
}
return &templateFile{t.data, file}, nil
}
hope this helps someone
@ninthclowd thanks for the contribution! I have just tested it in the latest version and it works great. It has been extremely helpful to inject some static environment values in the migrations.
