mattn
Problem with unstable go-sqlite3 version - 2.0.3
Hi,
We at the platform team of razorpay internally use bulk-insert which internally uses github.com/jinzhu/gorm v1.9.12 and creates an indirect dependency on github.com/mattn/go-sqlite3 v2.0.3.
Reference :- https://github.com/t-tiger/gorm-bulk-insert/blob/master/go.mod#L14
Now, this setup was working fine for us till last week and suddenly we had a failure in our docker build for github actions citing
github.com/mattn/[email protected]+incompatible: unknown revision v2.0.3
We also saw one of your issues where you mentioned versions 2.0+ were unstable - issue
Did you stop supporting 2.0.3 version suddenly in the last 7-10 days. We were wondering what caused the breakage of the build from our side. It would be helpful if you could throw some light on this issue.
Thanks, Shriram
I have the same problem with @Shriram-RP, but everything is going well after changing the GOPROXY env from direct to proxy.golang.org
Before changing the GOPROXY to proxy.golang.org
foo@bar golang-dir % go mod tidy
go: github.com/mattn/[email protected]+incompatible: reading github.com/mattn/go-sqlite3/go.mod at revision v2.0.1: unknown revision v2.0.1
go: downloading github.com/mattn/go-sqlite3 v2.0.1+incompatible
go: github.com/mattn/[email protected]+incompatible: reading github.com/mattn/go-sqlite3/go.mod at revision v2.0.1: unknown revision v2.0.1
Changing the GOPROXY to proxy.golang.org
foo@bar golang-dir % export GOPROXY=https://proxy.golang.org
After changing the GOPROXY to proxy.golang.org
foo@bar golang-dir % go mod tidy
go: downloading github.com/mattn/go-sqlite3 v2.0.1+incompatible
What version of Go are you using? Also, what is the output of
go env?
Version - 1.17
go env output :-
GO111MODULE="" GOARCH="arm64" GOBIN="" GOCACHE="/Users/{name}/Library/Caches/go-build" GOENV="/Users/{name}/Library/Application Support/go/env" GOEXE="" GOEXPERIMENT="" GOFLAGS="" GOHOSTARCH="arm64" GOHOSTOS="darwin" GOINSECURE="" GOMODCACHE="/Users/{name}/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="darwin" GOPATH="/Users/{name}/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/opt/homebrew/Cellar/go/1.17/libexec" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/opt/homebrew/Cellar/go/1.17/libexec/pkg/tool/darwin_arm64" GOVCS="" GOVERSION="go1.17" GCCGO="gccgo" AR="ar" CC="clang" CXX="clang++" CGO_ENABLED="1" GOMOD="/Users/{name}go/src/github.com/razorpay/reminders/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/qs/9hmssmm50p7djrwxsww6nfkm0000gq/T/go-build1013432823=/tmp/go-build -gno-record-gcc-switches -fno-common"
One way to fix this is adding the below code to the go.mod file
exclude github.com/mattn/go-sqlite3 v2.0.3+incompatible
This is not an issue of go-sqlite3. v2 was an accident as written in README.md
NOTE: The increase to v2 was an accident. There were no major changes or features.
@mattn I wonder if this would be a reason to use a retraction and just retag v2: https://golang.org/ref/mod#go-mod-file-retract
The impact of this is a bit broad. Large projects seem to have been impacted by this as well as small random odds and ends. Notice that tekton/cli was affected when I was trying to import it into my project. Also searching my cache I see gorm managed to get a commit with a v2 tag.
Can't imagine what else this might have landed in. A retraction seems like it would prevent tons of people from having to stuff exclude and replace into their module files, if I understand the mechanics of it correctly.
[protosam@nullhost]$ go mod init placebo
go: creating new go.mod: module placebo
[protosam@nullhost]$ go get github.com/tektoncd/[email protected]
go get: github.com/tektoncd/[email protected] updating to
github.com/tektoncd/[email protected] requires
github.com/tektoncd/hub/[email protected] requires
github.com/go-testfixtures/testfixtures/[email protected] requires
github.com/mattn/[email protected]+incompatible: reading github.com/mattn/go-sqlite3/go.mod at revision v2.0.2: unknown revision v2.0.2
[protosam@nullhost]$ grep -ri 'mattn/go-sqlite3 v2' ~/go/pkg/mod/cache
~/go/pkg/mod/cache/download/github.com/go-testfixtures/testfixtures/v3/@v/v3.2.0.mod: github.com/mattn/go-sqlite3 v2.0.2+incompatible
~/go/pkg/mod/cache/download/github.com/jinzhu/gorm/@v/v1.9.12.mod: github.com/mattn/go-sqlite3 v2.0.1+incompatible
If once you clean module cache on your environment, vertion 1.14 shold be used in next time.
That's not the case. People will be stumbling onto this for a while. go-sqlite3 is a very wide spread dependency and gets added into other projects during both manual and automated tasks.
Commits like this and this are actively referenced in projects today.
There are 39,000 files that come up with I search to get an idea of how wide spread this is.
@protosam From what I understand, retracting the v2 tags won't resolve any of the existing issues. It just stops the issue from continuing to grow. If I have a dependency on some other module that in turn has a dependency on one of the broken v2 tags of this library, then go mod will still attempt to download that version. And unless you are using proxy.golang.org (or similar), it will fail. The only time retraction matters is when someone is explicitly upgrading their dependencies.
In other words, even if a retraction is published, someone would still have to go to every one of the broken dependents and fix their go.mod to reference a valid release. Then all affected developers have to go and update to the newer versions of these modules.
@mattn I'm guessing the reason this just become an issue recently is because the v2 tags were deleted from this repo. Is that correct? If so, would you be able to re-create them for the same commits? That way people using "direct" at least won't get failed builds. (This is orthogonal to retracting the v2 releases.)
@mattn I'm looking into this. I expect to find it in a go.sum file somewhere.
Google had them cached in sum.golang.org and also they're still listed at pkg.go.dev.
v2.0.0+incompatible v2.0.1+incompatible v2.0.2+incompatible v2.0.3+incompatible
I'm actually a bit confused on the serialization. Saying what I got in case someone else is able to understand deserializing this faster than I can.
v2.0.0 was not a tag. It was branch name.
Last commit of v2.0.0 branch was
https://github.com/mattn/go-sqlite3/pull/611/commits/e904eb404c8ead3ded9e8fe4b00cc473ca2ce6c0
v2.0.1 https://web.archive.org/web/20201202062638/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.1 590d44c02bca83987d23f6eab75e6d0ddf95f644
v2.0.2 https://web.archive.org/web/20201202062638/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.2 53cff3fcebd1e177d04129dc131523635bc45d3d
v2.0.3 https://web.archive.org/web/20201202060439/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.3 9bdaffc12bf8be15afceb51bb60851edd4afdff5
v2.0.4 https://web.archive.org/web/20201202060440/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.4 db4c9426f851c76327db11e1b5301e520e0e048b
v2.0.5 https://web.archive.org/web/20201202060440/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.5 db4c9426f851c76327db11e1b5301e520e0e048b
v2.0.6 https://web.archive.org/web/20201202060441/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.6 baaf8a978416040e7f2d00ac36e345098d0588d8
So you (who used v2.0.X) should modify go.mod to switch v1.4.X.
require github.com/mattn/go-sqlite3 v1.14.9
The thousands of forks and archive.org are pretty useful, these commits look good to me and below I've tested them. They work as I expect (they validated against sum.golang.org).
I did some additional testing to see how the retraction will work. I don't think it works against legacy versioning. Ideally if it worked, we would be capable of getting a message out to the end users so they know they're using bad versions and later these versions can be gracefully claimed for reuse.
No idea where to go with that discovery though. =/
So v2.0.0, v2.0.1, v2.0.2, and v2.0.3 work because the +incompatible tag and there is not a go.mod file (legacy versioning).
Tags v2.0.4, v2.0.5, and v2.0.6 do not seem to be needed. These have go.mod versioning and yield errors that people would have to do more work than necessary to work around. They also don't show up in sum.golang.org and are not listed on pkg.go.dev. Seems likely that no one would miss these tags. This leads me to believe they've not been in use.
Started with an empty project and tested getting a module that has a lower dependent that requires v2.0.2.
[protosam@nullhost]$ go mod init placebo
go: creating new go.mod: module placebo
[protosam@nullhost]$ go get github.com/tektoncd/[email protected]
go get: added github.com/tektoncd/cli v0.21.0
[protosam@nullhost]$ go mod verify
all modules verified
Below I tested against v2.0.0 - v2.0.6
[protosam@nullhost]$ go mod download github.com/mattn/[email protected]+incompatible
[protosam@nullhost]$ go mod tidy
[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go get: upgraded github.com/mattn/go-sqlite3 v1.14.9 => v2.0.0+incompatible
[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go get: upgraded github.com/mattn/go-sqlite3 v2.0.0+incompatible => v2.0.1+incompatible
[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go get: upgraded github.com/mattn/go-sqlite3 v2.0.1+incompatible => v2.0.2+incompatible
[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go: downloading github.com/mattn/go-sqlite3 v2.0.3+incompatible
go get: upgraded github.com/mattn/go-sqlite3 v2.0.2+incompatible => v2.0.3+incompatible
[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go get: github.com/mattn/[email protected]+incompatible: invalid version: +incompatible suffix not allowed: module contains a go.mod file, so semantic import versioning is required
[protosam@nullhost]$ go get github.com/mattn/[email protected]
go get: github.com/mattn/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2
[protosam@nullhost]$ go get github.com/mattn/[email protected]
go get: github.com/mattn/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2
[protosam@nullhost]$ go get github.com/mattn/[email protected]
go get: github.com/mattn/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2
@mattn I'd probably pin this issue while it's affecting people otherwise they are going to open new ones as they rarely look at slightly older issues.
I'd probably reach out to the Go team at this point to see if they have any ideas to stop the spread and fix the problem (eg. remove these entries from sumdb? Dunno if that's possible/good idea)
Retracting would probably mean that you can't use these tags in the future.
This issue is already reported to Go team. https://github.com/golang/go/issues/35732
@mattn as far as I can tell that issue was closed with working as intended. I was rather suggesting to contact them about this specific case, hoping that they could remove the offending entries from the sumdb/proxy.
One way to fix this is adding the below code to the go.mod file
exclude github.com/mattn/go-sqlite3 v2.0.3+incompatible
Thank you! This fixed my problem. When I ran go get github.com/mattn/[email protected], it downgraded another library I'm using (ory/fosite) from 0.40.2 to 0.35.1. I then added this line to my go.mod and ran go get github.com/ory/[email protected]. This upgraded the package without upgrading go-sqlite3 to the bad version.
I restored older v2.0.X tags. How goes good?
The tags are disappeared again :(
The 2.0.X was retracted.
https://pkg.go.dev/github.com/mattn/go-sqlite3?tab=versions
You may have already had go-sqlite3 version 2.0.X. Please update manually to 1.4.X.
@mattn Even though you retracted those versions, those tags should never be deleted, so that people can continue to download their dependencies, including for older versions of their projects.
@rittneje Yes. My understanding is that once people manually revert to 1.4.X, they will not be upgraded thereafter since 2.X was retracted. I believe this is a problem that cannot be recovered automatically.
My point is that deleting those v2 tags breaks people's builds so we should not do that. We can only strongly discourage their use going forward (which the retraction helps).
But those tags were already released. So there are other libraries and projects that reference them. By deleting them, those versions cannot be built properly anymore (without using a Go module proxy), which was the original stated issue. We can tell people not to use v2 in new versions of their code anymore, but we should not cause the older/existing versions to break.
v2 was an accidental release because the Go mod incorrectly identified the branch name as a tag. https://github.com/mattn/go-sqlite3/pull/998 is a PR to add retract as a way to fix these problems.