Version 3.0.0 release

[mattes]

I just pushed version 3.0.0. I expect hiccups and questions. Please ping me anytime if you need help.

If you don't want to update to the latest version, you can continue to use v1.

[adrianduke]

We appear to be getting the following:

$ go get -v -u github.com/mattes/migrate
github.com/mattes/migrate (download)
# cd /.../go/src/github.com/mattes/migrate; git pull --ff-only
From https://github.com/mattes/migrate
 + 09640b1...f00c15d master     -> origin/master  (forced update)
 * [new branch]      v1         -> origin/v1
 * [new tag]         v1.3.1     -> v1.3.1
 * [new tag]         v3.0.0     -> v3.0.0
fatal: Not possible to fast-forward, aborting.
package github.com/mattes/migrate: exit status 128

[mattes]

That's expected. I forced push to master. At this point, v3 is not a drop in replacement for v1. Especially since not all drivers are available, yet.

In order to keep v1, you may want to fetch and checkout v1 branch.

Sorry for the inconvenience and breaking changes. If there is anything that would make your life easier, please let me know.

[b0bu]

Hey I think this is related, and I'm not sure how to get around it.

I'm using docker Notary in production and the official notary server/signer Dockerfile reference your github.com/mattes/migrate with go get. During a routine test on Friday it pulled the 'dev' version during image build and as result is broken as hell. Basically Notary is down for me because I can't get v3 to migrate the databases. At this point their migrate.sh script is exiting with 'no migration'.

Since I can't use go get to pull v1, and I can't seem to build it I thought I'd mention it. Any ideas

Update

It's not pretty, and if you or anyone has some elegant suggestions on syntax please advise (since I've been working with go since yesterday). Anyone still needing this to work with 1.3.1 feel free to add this to the Notary Dockerfiles.

# Dockerfile
# Install SQL DB migration tool
#RUN go get github.com/mattes/migrate
RUN git clone -b v1 https://github.com/mattes/migrate.git /go/src/github.com/mattes/migrate/
RUN go get -u -v github.com/mattes/migrate && \
    go build -tags 'mysql' -o /usr/local/bin/migrate github.com/mattes/migrate

I built it into its own image to test

docker run --rm -it --name migrate migrate:0.1 -version
1.3.1

Now docker stack up -c notary-stack.yml notary should work as long as your stack file is correct.

[morriswinkler-simple]

V1 is not go getable anymore.

Please try

go get gopkg.in/mattes/migrate.v1

I guess that broke a lot of stuff, at least for us, we currently fetch prebuild binaries as a workaround, would be nice so if go get for v1 would work again.

[ac-obair]

That's still returning 10 package dep errors:

package github.com/mattes/migrate/driver/bash: cannot find package "github.com/mattes/migrate/driver/bash" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/bash (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/bash (from $GOPATH)
package github.com/mattes/migrate/driver/cassandra: cannot find package "github.com/mattes/migrate/driver/cassandra" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/cassandra (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/cassandra (from $GOPATH)
package github.com/mattes/migrate/driver/crate: cannot find package "github.com/mattes/migrate/driver/crate" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/crate (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/crate (from $GOPATH)
...
...

The point of me pulling the v1 branch and then running go get was to get around that. Unless there's something I just don't know (likely)

[quater]

Same problem here go get gopkg.in/mattes/migrate.v1 tries to run import against paths that don't exist in master anymore.

https://github.com/mattes/migrate/blob/v1/main.go

For example:

github.com/mattes/migrate/driver/crate

should be

github.com/mattes/migrate/tree/v1/driver/crate

[swapii]

Backward compatibility of sources is one of primary feature in Go language. I think this is wrong to delete many files from library that used in many projects on the world.

If you need to develop new version of library you should fork repository and break compatibility in it.

Now I'm thinking about variants of building own project based on this library...

[gravis]

This fork should contains files (drivers) working with your project: https://github.com/db-journey/ It's a fork of the v1 repo.

[mattes]

I apologize for the inconvenience and I can understand everyones frustration over this. What can we do to make this easier?

I could update the import paths for the drivers so it's pointing to gopkg.in/mattes/migrate.v1. @morriswinkler-simplesurance @quater

It would make maintaining v1 harder though.

[jrock2468]

You could make the V1 version it's own repo "migrate" and a separate new repo "migrate-v3". A part of me hates this but since go get always pulls from the HEAD of the default branch in a repository, it would likely solve the immediate issues existing projects are experiencing.

[UnrealKazu]

Good job on the new version. I've got some remarks, though:

• There is no migration for the schema_migrations table if you're upgrading from v1 to v3. Meaning that any migration commands will fail because the table doesn't have a dirty column.
• No create command? I know it's trivial to create the files myself, but the command incorporates the timestamp which makes manually creating them bothersome.

[quater]

@mattes Thank you for developing and maintaining such a useful tool. Though, this version bump causes us some headache.

We are currently sitting between two chairs. Version 1 stopped working (not installable anymore) and it has been deprecated but on the other hand, we are currently not able to use Version 3 either since we have an existing database.

We are getting the below error, when we try to run version 3 by using our exiting database.

stderr: error: pq: column "dirty" does not exist in line 0: SELECT version, dirty FROM "schema_migrations" LIMIT 1

As @UnrealKazu highlighted, this fails due to a missing migration for the schema_migrations table. Speaking solely for ourselves, we are happy to move to Version 3. However it would be much appreciated by everyone if a migration is provided.

[mattes]

For Postgres and Mysql, I just added upgrade guides: https://github.com/mattes/migrate/tree/master/database/postgres#upgrading-from-v1 https://github.com/mattes/migrate/tree/master/database/mysql#upgrading-from-v1

This doesn't solve the go get v1 issue, yet. More updates on that in the next hours. Thanks for the patience everyone!

[mattes]

@UnrealKazu Re: no create command.

There are several questions around the expected behavior for create:

timestamp or number? _ migration_name _ extension?

To get the extension, in v1 the driver had to be loaded (https://github.com/mattes/migrate/issues/187).

I could see create output two files like this: timestamp_xxx.up.xxx timestamp_xxx.down.xxx

What are your thoughts?

[tfmertz]

I'm having trouble finding a way to implement the migrating similar to the old UpSync() method in v1. We were UpSyncing when the docker container was created to run the most recent migrations. This was working if there were or weren't pending migrations.

Currently, using Up() method gives a ErrNoChange if no migrations were run.

Is there a way to get similar functionality to UpSync with v3?

[mattes]

@tfmertz Just check for ErrNoChange. If you see it, you know you can continue.

[UnrealKazu]

@mattes Interesting points.

I would say that timestamp is the preferred prefix, as it results in a natural order without having to determine n in order to use n + 1 when you're using number as prefix. Also, what will you do if you delete a migration, as you would have a gap in your numbering, something you would not have with timestamps.

Regarding the extension: I can see why you don't want to be dependent on a driver for this, since it is only a create command and the driver will be deduced once you start to roll out your migrations. With this in mind, a generic extension would also suffice.

An extension matching the language of the content of the migration would be nice for some, but as mixing of drivers is not possible (I think?), all files would always have the same extension, and thus I don't think it would be insurmountable if they have a generic extension. Maybe we should move this discussion to a separate issue?

[morriswinkler-simple]

@mattes i would really appreciate if gopkg.in/mattes/migrate.v1 would work again,

Even if you would just make a new repro from v1 like github.com/mattes/migratev1 that would wok for me too.

One of the core ideas of go is to never break backward compatibility, this is not always that easy, but as Rob Pikes pointed out, simplicity is difficult.

[bilalaslamseattle]

@mattes +1 on making v1 work again. The v1 -> v3 breaking change is a HUGE headache for us.

[mattes]

I renamed all import paths (https://github.com/mattes/migrate/pull/212) to point to gopkg.in/mattes/migrate.v1. Getting this error. Can someone help?

go get -v gopkg.in/mattes/migrate.v1
Fetching https://gopkg.in/mattes/migrate.v1?go-get=1
Parsing meta tags from https://gopkg.in/mattes/migrate.v1?go-get=1 (status code 200)
get "gopkg.in/mattes/migrate.v1": found meta tag main.metaImport{Prefix:"gopkg.in/mattes/migrate.v1", VCS:"git", RepoRoot:"https://gopkg.in/mattes/migrate.v1"} at https://gopkg.in/mattes/migrate.v1?go-get=1
gopkg.in/mattes/migrate.v1 (download)
github.com/fatih/color (download)
github.com/mattes/migrate (download)
package github.com/mattes/migrate/driver/bash: cannot find package "github.com/mattes/migrate/driver/bash" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/bash (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/bash (from $GOPATH)
package github.com/mattes/migrate/driver/cassandra: cannot find package "github.com/mattes/migrate/driver/cassandra" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/cassandra (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/cassandra (from $GOPATH)
package github.com/mattes/migrate/driver/crate: cannot find package "github.com/mattes/migrate/driver/crate" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/crate (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/crate (from $GOPATH)
package github.com/mattes/migrate/driver/mysql: cannot find package "github.com/mattes/migrate/driver/mysql" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/mysql (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/mysql (from $GOPATH)
package github.com/mattes/migrate/driver/neo4j: cannot find package "github.com/mattes/migrate/driver/neo4j" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/neo4j (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/neo4j (from $GOPATH)
package github.com/mattes/migrate/driver/postgres: cannot find package "github.com/mattes/migrate/driver/postgres" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/postgres (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/postgres (from $GOPATH)
package github.com/mattes/migrate/driver/ql: cannot find package "github.com/mattes/migrate/driver/ql" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/ql (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/ql (from $GOPATH)
package github.com/mattes/migrate/driver/sqlite3: cannot find package "github.com/mattes/migrate/driver/sqlite3" in any of:
        /usr/local/go/src/github.com/mattes/migrate/driver/sqlite3 (from $GOROOT)
        /go/src/github.com/mattes/migrate/driver/sqlite3 (from $GOPATH)
package github.com/mattes/migrate/file: cannot find package "github.com/mattes/migrate/file" in any of:
        /usr/local/go/src/github.com/mattes/migrate/file (from $GOROOT)
        /go/src/github.com/mattes/migrate/file (from $GOPATH)
package github.com/mattes/migrate/migrate: cannot find package "github.com/mattes/migrate/migrate" in any of:
        /usr/local/go/src/github.com/mattes/migrate/migrate (from $GOROOT)
        /go/src/github.com/mattes/migrate/migrate (from $GOPATH)
package github.com/mattes/migrate/migrate/direction: cannot find package "github.com/mattes/migrate/migrate/direction" in any of:
        /usr/local/go/src/github.com/mattes/migrate/migrate/direction (from $GOROOT)
        /go/src/github.com/mattes/migrate/migrate/direction (from $GOPATH)
package github.com/mattes/migrate/pipe: cannot find package "github.com/mattes/migrate/pipe" in any of:
        /usr/local/go/src/github.com/mattes/migrate/pipe (from $GOROOT)
        /go/src/github.com/mattes/migrate/pipe (from $GOPATH)

[endophage]

In the spirit of things that would make my life easier, you should have ensured the binary name would stay the same. It's becoming reasonably common for go projects to have a cmd/ directory, with sub-directories that allow you to name the binary (because the dir name will be used as the binary name when one go get's or go install's). Therefore, what you currently have in ./cli/ should be in ./cmd/migrate/.

Also, please never force push master again.

[mattes]

I'm sorry for the confusion and inconvenience @endophage

Regarding the directories. Do you think it makes sense to refactor it?

[endophage]

Probably, cli is way too generic a binary name so I'm pretty sure nobody wants it to be named that. Moving to the cmd format allows one to e.g. go get -tags 'mysql file' github.com/mattes/migrate/cmd/migrate (unfortunately the -o flag on build isn't supported on get, but I guess technically you can do this with cli/ you just end up with the uninformative binary name)

[endophage]

Something else we've noticed is the version command now appears to return no migration or some such error on a brand new database. Previously I think it must have returned 0 or some other non-error when it could connect but no tables were found.

I think the new behaviour is probably more correct, it just caught us out as we have a script that tried to run version 30 times with 1 second sleeps in between until the database came up (it was a hack for the docker database container taking a little while to initialize). It would be great if changes like that were documented in release notes or "Upgrading to version X" type docs.

[mattes]

I'm trying my best to improve docs. Please feel free to PR things in as well. Every help is appreciated!

[morriswinkler-simple]

Regarding the problem of getting

go get gopkg.in/mattes/migrate.v1

that is because of the dependencies, github.com/mattes/migrate/driver/bash, for example, is pulled from master branch head, you would need to refactor this so it would import gopkg.in/mattes/migrate/driver/bash.v1, go get will always pull from master, gopkg.in is just a way to pull tags or branches, you would need to change all import path son the corresponding branches.

[morriswinkler-simple]

Ok, I saw you already did that, it still doesn't work. Cause the tag v1.3.1 is the one that gopk.in/mattes/migrate.v1 will use since it is the latest v1 release, please tag the head of branch v1 with a higher version 1 tag.

[mattes]

Oh, interesting. I see. Let me try that. Thanks for your help!

[morriswinkler-simple]

@mattes maybe you could refactor 1.3.1 to use import paths like gopkg.in/mattes/migrate.v1 if it is still compatible with v1, what do you thing

[morriswinkler-simple]

@mattes I can't believe you are giving up after al that, we are so close to making migrate v1 go getable again.