Too many dependencies

[dumindu]

Current dependencies:

require (
	dmitri.shuralyov.com/app/changes v0.0.0-20181114035150-5af16e21babb // indirect
	dmitri.shuralyov.com/service/change v0.0.0-20190203163610-217368fe4577 // indirect
	git.apache.org/thrift.git v0.12.0 // indirect
	github.com/Shopify/sarama v1.20.1 // indirect
	github.com/aws/aws-sdk-go v1.16.36 // indirect
	github.com/coreos/go-systemd v0.0.0-20190212144455-93d5ec2c7f76 // indirect
	github.com/cznic/b v0.0.0-20181122101859-a26611c4d92d // indirect
	github.com/cznic/fileutil v0.0.0-20181122101858-4d67cfea8c87 // indirect
	github.com/cznic/golex v0.0.0-20181122101858-9c343928389c // indirect
	github.com/cznic/internal v0.0.0-20181122101858-3279554c546e // indirect
	github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 // indirect
	github.com/cznic/sortutil v0.0.0-20181122101858-f5f958428db8 // indirect
	github.com/cznic/strutil v0.0.0-20181122101858-275e90344537 // indirect
	github.com/cznic/zappy v0.0.0-20181122101859-ca47d358d4b1 // indirect
	github.com/docker/distribution v2.7.1+incompatible // indirect
	github.com/edsrzf/mmap-go v1.0.0 // indirect
	github.com/fsouza/fake-gcs-server v1.5.0 // indirect
	github.com/go-ini/ini v1.41.0 // indirect
	github.com/go-logfmt/logfmt v0.4.0 // indirect
	github.com/gocql/gocql v0.0.0-20190208221138-c53c3654dc8a // indirect
	github.com/golang-migrate/migrate/v4 v4.2.4 // indirect
	github.com/golang/lint v0.0.0-20181217174547-8f45f776aaf1 // indirect
	github.com/google/pprof v0.0.0-20190208070709-b421f19a5c07 // indirect
	github.com/googleapis/gax-go v2.0.2+incompatible // indirect
	github.com/gopherjs/gopherjs v0.0.0-20181103185306-d547d1d9531e // indirect
	github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc // indirect
	github.com/grpc-ecosystem/grpc-gateway v1.7.0 // indirect
	github.com/jackc/pgx v3.3.0+incompatible // indirect
	github.com/kshvakov/clickhouse v1.3.5 // indirect
	github.com/mattn/go-sqlite3 v1.10.0 // indirect
	github.com/microcosm-cc/bluemonday v1.0.2 // indirect
	github.com/mongodb/mongo-go-driver v0.3.0 // indirect
	github.com/openzipkin/zipkin-go v0.1.5 // indirect
	github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90 // indirect
	github.com/prometheus/common v0.2.0 // indirect
	github.com/prometheus/procfs v0.0.0-20190209105433-f8d8b3f739bd // indirect
	github.com/russross/blackfriday v2.0.0+incompatible // indirect
	github.com/shurcooL/go v0.0.0-20190121191506-3fef8c783dec // indirect
	github.com/shurcooL/gofontwoff v0.0.0-20181114050219-180f79e6909d // indirect
	github.com/shurcooL/highlight_diff v0.0.0-20181222201841-111da2e7d480 // indirect
	github.com/shurcooL/highlight_go v0.0.0-20181215221002-9d8641ddf2e1 // indirect
	github.com/shurcooL/home v0.0.0-20190204141146-5c8ae21d4240 // indirect
	github.com/shurcooL/htmlg v0.0.0-20190120222857-1e8a37b806f3 // indirect
	github.com/shurcooL/httpfs v0.0.0-20181222201310-74dc9339e414 // indirect
	github.com/shurcooL/issues v0.0.0-20190120000219-08d8dadf8acb // indirect
	github.com/shurcooL/issuesapp v0.0.0-20181229001453-b8198a402c58 // indirect
	github.com/shurcooL/notifications v0.0.0-20181111060504-bcc2b3082a7a // indirect
	github.com/shurcooL/octicon v0.0.0-20181222203144-9ff1a4cf27f4 // indirect
	github.com/shurcooL/reactions v0.0.0-20181222204718-145cd5e7f3d1 // indirect
	github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect
	github.com/shurcooL/webdavfs v0.0.0-20181215192745-5988b2d638f6 // indirect
	github.com/smartystreets/assertions v0.0.0-20190215210624-980c5ac6f3ac // indirect
	github.com/smartystreets/goconvey v0.0.0-20181108003508-044398e4856c // indirect
	go.opencensus.io v0.19.0 // indirect
	go4.org v0.0.0-20181109185143-00e24f1b2599 // indirect
	golang.org/x/build v0.0.0-20190215225244-0261b66eb045 // indirect
	golang.org/x/exp v0.0.0-20190212162250-21964bba6549 // indirect
	golang.org/x/oauth2 v0.0.0-20190212230446-3e8b2be13635 // indirect
	golang.org/x/perf v0.0.0-20190124201629-844a5f5b46f4 // indirect
	golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a // indirect
	golang.org/x/tools v0.0.0-20190214204934-8dcb7bc8c7fe // indirect
	google.golang.org/genproto v0.0.0-20190215211957-bd968387e4aa // indirect
	google.golang.org/grpc v1.18.0 // indirect
	gopkg.in/ini.v1 v1.41.0 // indirect
	honnef.co/go/tools v0.0.0-20190215041234-466a0476246c // indirect
	sourcegraph.com/sqs/pbtypes v1.0.0 // indirect
)

Don't you think that this is too much for a db migration package?

Compared to other alternatives, this is too much

• ex. pressly/goose; only relative db drivers, even those can be reduced from binary via build -tags=

require (
	github.com/go-sql-driver/mysql v1.4.1 // indirect
	github.com/lib/pq v1.0.0 // indirect
	github.com/mattn/go-sqlite3 v1.10.0 // indirect
	github.com/pressly/goose v2.4.5+incompatible // indirect
	github.com/ziutek/mymysql v1.5.4 // indirect
)

[dhui]

This is probably due to migrate supporting more DBs than pressly/goose.

go mod tidy will add all dependencies regardless of build tag, etc Not all of these dependencies will be used in your final build.

[dhui]

Related issue: https://github.com/golang-migrate/migrate/issues/60

[dumindu]

But why we need openzipkin, prometheus like packages? Also github.com/shurcooL/home is the repo for Dmitri Shuralyov's personal website. Looks like something is very wrong.

[dhui]

go mod why is your friend

You may see some packages in go.sum, this explains why

Also, the go.mod snippet you originally provided doesn't look like the one for migrate...

[dumindu]

what I did/ steps to reproduce

• mkdir testpack && cd testpack
• go mod init testpack
• go get -u github.com/golang-migrate/migrate/v4

[dhui]

Looks like that's your own go.mod file... Check the module directive at the top of the file...

[dumindu]

@dhui The list I mentioned above is dependencies added for github.com/golang-migrate/migrate/v4 for an empty project. You also can reproduce this via above steps :)

[dhui]

Unfortunately, we can't reduce the number of dependencies in go.mod since they're used by the various supported source and db drivers.

Run go mod tidy after your steps in your test package to prune unused packages.

[sagikazarmark]

While I don't think there is such thing as "too much dependencies" (if they are needed, they are needed), the number of dependencies can make builds huge on the other hand.

Given golang-migrate has it's own organization, I would consider cutting the library into smaller pieces (modules) (eg. one for each driver and a separate one for the cmd).

It could ease the pain of users, but it would probably make development slightly harder.

Currently that's the recommendation you will get from go modules experts as a solution to this problem.

[dhui]

I would consider cutting the library into smaller pieces (modules) (eg. one for each driver and a separate one for the cmd). It could ease the pain of users, but it would probably make development slightly harder.

The base library, CLI, and drivers (source and database) are already separate packages. What's to be gained by making them separate modules? Could you be more specific about the pain points splitting into separate modules would alleviate?

[sagikazarmark]

Currently regardless of which package you import in a project, go modules will download all transitive dependencies, whether they are going to be compiled into the final binary or not, so separate packages make the work easier for the linker, but the amount of downloaded dependencies will still be the same. By making separate modules, you can limit these dependencies to the absolute minimum.

IMHO the problem of excessive amount of dependencies is obvious: slower CI, more downloaded stuff then necessary, etc.

I'm not saying you should do this, all I'm saying is if this is a problem and you want to solve it, splitting into separate modules can be a solution.

[dumindu]

While I don't think there is such thing as "too much dependencies" (if they are needed, they are needed)

@sagikazarmark According to the dependency list(on go.mod) which is generated via go get -u github.com/golang-migrate/migrate/v4, this package requires even github.com/shurcooL/home (Dmitri Shuralyov's personal website). This is only one dependency which might not needed or something is wrong if any dependency of this depends on someones' personal website.

[sagikazarmark]

I guess it's a mistake in a transitive dependency, not in golang-migrate itself. Go mod installs all transitive dependencies and test dependencies of all transitive dependencies as well.

[dhui]

IMHO the problem of excessive amount of dependencies is obvious: slower CI, more downloaded stuff then necessary, etc. I'm not saying you should do this, all I'm saying is if this is a problem and you want to solve it, splitting into separate modules can be a solution.

@sagikazarmark Thanks for the stating an advantage of using separate modules. At this time, extraneous transitive dependency downloads is not a concern for this project. Maybe the go tool chain will improve this aspect of dependency management later...

[sagikazarmark]

I believe it will, although not the go tool per se: https://blog.golang.org/modules2019#TOC_6.

Nevertheless, I'm going to link this issue to another one where I try to keep track of the problem (a few libraries of mine suffer from the same issue). I'm also a regular complainer on the Gophers slack #modules channel, so I will also gather information from there and record in the issue.

[thepudds]

Also github.com/shurcooL/home is the repo for Dmitri Shuralyov's personal website. Looks like something is very wrong.

A healthy chunk of of the dependencies cited above seem to be coming in via golang.org/x/..., such as via golang/x/build:

golang.org/x/[email protected] 
github.com/shurcooL/[email protected]

The various golang.org/x repos have been updated recently to (a) have go.mod files and (b) to cut down on the number of extra dependencies, especially for golang.org/x/build.

Updating to the latest golang.org/x repos would likely help. That can be down manually, or via something like the following:

$ for x in $(go list  -f '{{.Path}}' -m golang.org/x/...); do go get $x@latest; done

And then run go mod tidy to eliminate from go.mod and go.sum any dependencies that are no longer needed.

[thepudds]

@dumindu

According to the dependency list(on go.mod) which is generated via go get -u github.com/golang-migrate/migrate/v4, ...

Note that you should consider whether or not you wanted to include a -u there.

From the "How to upgrade and downgrade dependencies" section of the "Modules" wiki:

go get foo updates to the latest version with a semver tag, or the the latest known commit if there are no semver tags. go get foo is equivalent to go get foo@latest — in other words, @latest is the default if no @ version is specified.

Note that a -u in go get -u foo and go get -u foo@latest also upgrades all the direct and indirect dependencies of foo. A common starting point is instead go get foo or go get foo@latest without a -u (and after things are working, consider go get -u=patch foo, go get -u foo, go get -u, or go get -u=patch).

[hbobenicio]

Currently regardless of which package you import in a project, go modules will download all transitive dependencies, whether they are going to be compiled into the final binary or not, so separate packages make the work easier for the linker, but the amount of downloaded dependencies will still be the same. By making separate modules, you can limit these dependencies to the absolute minimum.

IMHO the problem of excessive amount of dependencies is obvious: slower CI, more downloaded stuff then necessary, etc.

I'm not saying you should do this, all I'm saying is if this is a problem and you want to solve it, splitting into separate modules can be a solution.

I agree with the suggestion of @sagikazarmark about the modularization. If I need to use migrations for my PG database, it feels just awkward to have dependencies (either transitive or not) for things like github.com/aws/aws-sdk-go or github.com/denisenkom/go-mssqldb that are just relevant to the respective datasources, which does not match the actual user datasources.

Maybe the driver pattern could come in handy here in order to separate interfaces from datasource specific implementations, allowing modularization. This is the approach that others like (db-migrate)[https://github.com/db-migrate/node-db-migrate] (which would require db-migrate and db-migrate-pg dependencies for the user).

[dhui]

@hbobenicio AFAIK, the "extraneous" (e.g. unused) dependencies won't affect your builds.

That being said, we may split drivers into their own modules so we can better adhere to SemVer and have different release cycles/cadences.

[majewsky]

the "extraneous" (e.g. unused) dependencies won't affect your builds.

It does affect the build time because it spends precious minutes finding and downloading all these junk dependencies. That may not be a big problem when your pipeline takes hours anyway, but I use Go precisely because everything is fast. An additional minute of slowdown is really painful when your entire pipeline (from git commit to prod) is 5 minutes.

I will definitely hold off upgrading to migrate v4 until this issue is sorted out.

[thepudds]

It does affect the build time because it spends precious minutes finding and downloading all these junk dependencies.

@majewsky If you haven't already, you should try with Go 1.13.

Downloading dependencies is also often 5x - 10x faster in Go 1.13 due to visiting fewer dependencies, and due to defaulting to a GOPROXY (which is more efficient than using VCS as a coarse-grained transport).

Note: If you have private code, you'll likely want to configure the GOPRIVATE setting (such as go env -w GOPRIVATE=*.corp.com) as described in the 1.13 release notes and elsewhere.

[majewsky]

The 1.13 release is what prompted me to bump my deps in the first place. And I don't have private repos.

[sagikazarmark]

Chances are go mod download will only be slightly faster, as it still downloads everything. So locally and in CI do not use go mod download. In docker you don't really have a choice though. 😕

I ended up splitting up my adapter type packages into smaller packages. Not sure if it's better (yet), but solved my dependency issues.

[amadsen]

We ran in to this issue this week because, as @sagikazarmark points out, go mod download downloads everything. For our docker based build this results in a layer in our docker image that is over 500MB. The build would technically work fine, but we then have issues pushing this large layer to a private docker registry because it is too large.

We've worked around this by doing a multi-stage docker build so that the deployed image does not contain the go mod download layer, but it would be much appreciated if my project didn't indirectly depend on every library needed for every supported database and source for migrations.

[sagikazarmark]

Unfortunately, this is a Go modules issue. The alternative would be splitting the drivers into separate go modules. Hashicorp vault project is a good example.

[veqryn]

I'd like to make use of some of the code in the source directory, without having 100s of dependencies show up and slow down the go mod commands and my ide. May I suggest multiple modules (in the same repo, or split to separate repos), where you have a "core" migrate module that has all main code without all the database dependencies. Then one module per database family. Then, optionally, one module that imports everything, for people who are lazy and don't want to find which specific one they need (or they have multiple databases and need it to support everything).

[thepudds]

It looks like the go.mod file here indicates go 1.16:

https://github.com/golang-migrate/migrate/blob/4ba695707abdb86fc76501cd96c82abca280dd28/go.mod#L71

When this project is ready to move that to go 1.17 and do go mod tidy with the 1.17 go toolchain, I think the situation described above will materially improve for go 1.17+ consumers.

The 1.17 release notes have more details about the new “pruned module graphs” behavior in go 1.17 modules:

https://go.dev/doc/go1.17#go-command

@veqryn if you are curious, you could try that with a local cloned copy of this repo, and point your consumer towards it with a local file based replace directive in your consumer’s go.mod file.

[SuperSandro2000]

This can be closed with the latest release and specifically #815

[jetexe]

What do you think about AWS SDK approach? Could we put mod file into each datasource and database package? It will reduce transitive dependencies to minimum