x icon indicating copy to clipboard operation
x copied to clipboard
verified publisher icon charmbracelet

pkg checksum mismatch

Open LeeBrotherston opened this issue 1 month ago • 2 comments

Description

On multiple versions of the pkg I get errors such as (obviously different values for different versions, but you get the idea):

verifying github.com/charmbracelet/x/[email protected]: checksum mismatch
	downloaded: h1:iXAC8SyMQDJgtcz9Jnw+HU8WMEctHzoTAETIeA3JXMk=
	go.sum:     h1:OSaQLe/B39MuYa0QwsJXJp378jUzFTqxMY99Zlz5U88=

Often this will occur when development takes place locally and then a github action tries to build my code.

I notice that this has occurred a few times in the past based on issues, so I wonder if there is a systemic issue in the release process rather than a one off glitch?

Previously when I have seen this (in other repos), it was the case that the release process caused the package to change during release or reuse version numbers so that sum.golang.org cached an entry and then a valid but different package content is served from there onwards. Then the users experiencing the issue depended on if they used sumdb or not.

Version

v0.10.2 & v0.11.1 (maybe others?)

Environment

macos / ubuntu

LeeBrotherston avatar Nov 18 '25 14:11 LeeBrotherston

i've seen this happen when having goprivate set - maybe that's your case?

caarlos0 avatar Nov 18 '25 15:11 caarlos0

@caarlos0 I'm afraid not, no...

$ go env | grep -F PRIVATE
GOPRIVATE=''

LeeBrotherston avatar Nov 18 '25 15:11 LeeBrotherston

Same issue when I try to build glow with ansi v0.10.1.

❯ go mod tidy
go: downloading github.com/charmbracelet/glamour v0.10.0
go: downloading github.com/charmbracelet/x/ansi v0.10.1
go: downloading github.com/kylelemons/godebug v1.1.0
go: downloading github.com/stretchr/testify v1.11.1
go: downloading github.com/kr/pretty v0.3.1
go: downloading github.com/frankban/quicktest v1.14.6
go: downloading github.com/rogpeppe/go-internal v1.12.0
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/google/go-cmp v0.6.0
verifying github.com/charmbracelet/x/[email protected]: checksum mismatch
	downloaded: h1:LT77A3bpevRD0yZ5NDR5nonS7N83mxzzGwuZcTGezLE=
	go.sum:     h1:rL3Koar5XvX0pHGfovN03f5cxLbCF2YvLeyz7D2jVDQ=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'

wold9168 avatar Nov 25 '25 01:11 wold9168

i can't reproduce this in a clean env.

my guesses:

  • make sure GOPROXY and GOPRIVATE are the default values
  • remove your go mod cache stuff (entire GOPATH to be sure)

if you can reproduce this in an empty environment (e.g. a docker container), please share how. Right now I can't reproduce this.

$ docker run -it --rm golang:1.25-alpine
Unable to find image 'golang:1.25-alpine' locally
1.25-alpine: Pulling from library/golang
6b59a28fa201: Already exists
17f1ad36a5a8: Pull complete
6e0cd0526a95: Pull complete
b381f5933278: Pull complete
4f4fb700ef54: Pull complete
Digest: sha256:d3f0cf7723f3429e3f9ed846243970b20a2de7bae6a5b66fc5914e228d831bbb
Status: Downloaded newer image for golang:1.25-alpine
/go # apk add -U git
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/community/aarch64/APKINDEX.tar.gz
(1/12) Installing brotli-libs (1.1.0-r2)
(2/12) Installing c-ares (1.34.5-r0)
(3/12) Installing libunistring (1.3-r0)
(4/12) Installing libidn2 (2.3.7-r0)
(5/12) Installing nghttp2-libs (1.65.0-r0)
(6/12) Installing libpsl (0.21.5-r3)
(7/12) Installing zstd-libs (1.5.7-r0)
(8/12) Installing libcurl (8.14.1-r2)
(9/12) Installing libexpat (2.7.3-r0)
(10/12) Installing pcre2 (10.46-r0)
(11/12) Installing git (2.49.1-r0)
(12/12) Installing git-init-template (2.49.1-r0)
Executing busybox-1.37.0-r19.trigger
OK: 20 MiB in 29 packages
/go # git clone https://github.com/charmbracelet/glow
Cloning into 'glow'...
remote: Enumerating objects: 4210, done.
remote: Counting objects: 100% (429/429), done.
remote: Compressing objects: 100% (87/87), done.
remote: Total 4210 (delta 390), reused 343 (delta 342), pack-reused 3781 (from 3)
Receiving objects: 100% (4210/4210), 3.00 MiB | 8.83 MiB/s, done.
Resolving deltas: 100% (2865/2865), done.
/go # cd glow
/go/glow # go mod tidy
go: downloading github.com/charmbracelet/glamour v0.10.0
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/charmbracelet/lipgloss v1.1.1-0.20250404203927-76690c660834
go: downloading github.com/caarlos0/env/v11 v11.3.1
go: downloading github.com/charmbracelet/log v0.4.2
go: downloading github.com/charmbracelet/x/editor v0.1.0
go: downloading github.com/muesli/go-app-paths v0.2.2
go: downloading github.com/muesli/mango-cobra v1.3.0
go: downloading github.com/muesli/roff v0.1.0
go: downloading github.com/spf13/cobra v1.10.1
go: downloading github.com/spf13/viper v1.21.0
go: downloading golang.org/x/sys v0.38.0
go: downloading golang.org/x/term v0.37.0
go: downloading github.com/atotto/clipboard v0.1.4
go: downloading github.com/charmbracelet/bubbles v0.21.0
go: downloading github.com/charmbracelet/bubbletea v1.3.10
go: downloading github.com/dustin/go-humanize v1.0.1
go: downloading github.com/mattn/go-runewidth v0.0.19
go: downloading github.com/muesli/reflow v0.3.0
go: downloading github.com/fsnotify/fsnotify v1.9.0
go: downloading github.com/muesli/gitcha v0.3.0
go: downloading github.com/muesli/termenv v0.16.0
go: downloading github.com/sahilm/fuzzy v0.1.1
go: downloading golang.org/x/text v0.31.0
go: downloading github.com/go-logfmt/logfmt v0.6.0
go: downloading golang.org/x/exp v0.0.0-20240604190554-fc45aab8b7f8
go: downloading github.com/inconshreveable/mousetrap v1.1.0
go: downloading github.com/spf13/pflag v1.0.10
go: downloading github.com/yuin/goldmark v1.7.8
go: downloading github.com/yuin/goldmark-emoji v1.0.5
go: downloading github.com/charmbracelet/x/ansi v0.10.1
go: downloading github.com/charmbracelet/x/cellbuf v0.0.13
go: downloading github.com/rivo/uniseg v0.4.7
go: downloading github.com/alecthomas/chroma/v2 v2.14.0
go: downloading github.com/charmbracelet/x/exp/slice v0.0.0-20250327172914-2fdc97757edf
go: downloading github.com/microcosm-cc/bluemonday v1.0.27
go: downloading github.com/go-viper/mapstructure/v2 v2.4.0
go: downloading github.com/sagikazarmark/locafero v0.11.0
go: downloading github.com/spf13/afero v1.15.0
go: downloading github.com/spf13/cast v1.10.0
go: downloading github.com/sabhiram/go-gitignore v0.0.0-20180611051255-d3107576ba94
go: downloading github.com/aymanbagabas/go-osc52/v2 v2.0.1
go: downloading github.com/lucasb-eyer/go-colorful v1.2.0
go: downloading github.com/mattn/go-isatty v0.0.20
go: downloading github.com/clipperhouse/uax29/v2 v2.2.0
go: downloading github.com/charmbracelet/x/term v0.2.1
go: downloading github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f
go: downloading github.com/mattn/go-localereader v0.0.1
go: downloading github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6
go: downloading github.com/muesli/cancelreader v0.2.2
go: downloading github.com/charmbracelet/colorprofile v0.2.3-0.20250311203215-f60798e515dc
go: downloading github.com/dlclark/regexp2 v1.11.0
go: downloading github.com/aymerick/douceur v0.2.0
go: downloading golang.org/x/net v0.40.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/pelletier/go-toml/v2 v2.2.4
go: downloading go.yaml.in/yaml/v3 v3.0.4
go: downloading github.com/muesli/mango v0.2.0
go: downloading github.com/muesli/mango-pflag v0.1.0
go: downloading github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8
go: downloading github.com/stretchr/testify v1.11.1
go: downloading github.com/charmbracelet/x/exp/golden v0.0.0-20241011142426-46044092ad91
go: downloading github.com/kylelemons/godebug v1.1.0
go: downloading github.com/alecthomas/assert/v2 v2.7.0
go: downloading github.com/frankban/quicktest v1.14.6
go: downloading github.com/alecthomas/repr v0.4.0
go: downloading github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e
go: downloading github.com/gorilla/css v1.0.1
go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c
go: downloading github.com/hexops/gotextdiff v1.0.3
go: downloading github.com/kr/pretty v0.3.1
go: downloading github.com/google/go-cmp v0.6.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading github.com/rogpeppe/go-internal v1.12.0
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/aymanbagabas/go-udiff v0.2.0
/go/glow # go build .
/go/glow #

caarlos0 avatar Nov 25 '25 19:11 caarlos0 

❯ go mod tidy
go: downloading github.com/charmbracelet/glamour v0.10.0
go: downloading github.com/charmbracelet/x/ansi v0.10.1
go: downloading github.com/stretchr/testify v1.11.1
go: downloading github.com/google/go-cmp v0.6.0
go: downloading github.com/rogpeppe/go-internal v1.12.0
go: downloading github.com/kr/text v0.2.0
verifying github.com/charmbracelet/x/[email protected]: checksum mismatch
        downloaded: h1:LT77A3bpevRD0yZ5NDR5nonS7N83mxzzGwuZcTGezLE=
        go.sum:     h1:rL3Koar5XvX0pHGfovN03f5cxLbCF2YvLeyz7D2jVDQ=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.
❯ printenv |grep GO
❯ go env GOPRIVATE

❯ go env GOPROXY
direct

I will try to reproduce this in my docker later.

wold9168 avatar Nov 25 '25 19:11 wold9168

git clone https://github.com/charmbracelet/glow

I ran the exact same command:

git clone https://github.com/charmbracelet/glow

Everything worked perfectly—no weird behavior at all. All commands executed without issue. But why?

Could this be related to the Go runtime?

  • Go runtime that works fine:

    /go/glow # go version
go version go1.25.4 linux/amd64
/go/glow # go version
go version go1.24.10 linux/amd64

  • Go runtime that behaves poorly (fedora42):

    ❯ go version
go version go1.24.9 linux/amd64

I’ve tried to reproduce this issue in a clean fedora:42 container, but it still doesn’t occur.

So, aside from GOPROXY and GOPRIVATE, what else could affect go mod tidy?

I've just cleaned the mod cache on the problem machine with go clean -modcache.

wold9168 avatar Nov 25 '25 19:11 wold9168

Hope this helps!

❯ go clean -modcache
❯ go mod tidy -x|grep ansi
mkdir -p /home/wold9168/go/pkg/mod/cache/vcs # git3 https://github.com/muesli/ansi
mkdir -p /home/wold9168/go/pkg/mod/cache/vcs/a36d7b5dab9137cbc3f11cf216a5abe1aff9ab84a10ff42a1ba58068a4dc2c94 # git3 https://github.com/muesli/ansi
cd /home/wold9168/go/pkg/mod/cache/vcs/a36d7b5dab9137cbc3f11cf216a5abe1aff9ab84a10ff42a1ba58068a4dc2c94; git remote add origin -- https://github.com/muesli/ansi
0.015s # cd /home/wold9168/go/pkg/mod/cache/vcs/a36d7b5dab9137cbc3f11cf216a5abe1aff9ab84a10ff42a1ba58068a4dc2c94; git remote add origin -- https://github.com/muesli/ansi
go: downloading github.com/charmbracelet/x/ansi v0.10.1
go: downloading github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6
cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/ansi/v0.10.1:refs/tags/ansi/v0.10.1
3.251s # cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/ansi/v0.10.1:refs/tags/ansi/v0.10.1
cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' refs/tags/ansi/v0.10.1 --
0.004s # cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' refs/tags/ansi/v0.10.1 --
cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git cat-file blob 38fb69db254fb15a39427143e76768fe176e8b9b:ansi/go.mod
0.004s # cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git cat-file blob 38fb69db254fb15a39427143e76768fe176e8b9b:ansi/go.mod
cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git cat-file blob 38fb69db254fb15a39427143e76768fe176e8b9b:ansi/go.mod
0.003s # cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git cat-file blob 38fb69db254fb15a39427143e76768fe176e8b9b:ansi/go.mod
cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git cat-file blob 38fb69db254fb15a39427143e76768fe176e8b9b:ansi/go.mod
0.004s # cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git cat-file blob 38fb69db254fb15a39427143e76768fe176e8b9b:ansi/go.mod
cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 38fb69db254fb15a39427143e76768fe176e8b9b -- ansi
1.471s # cd /home/wold9168/go/pkg/mod/cache/vcs/978544cfd34d5e1ff7daa7a515f2bce075f3b9d21cd2b201e96da1995c29b127; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 38fb69db254fb15a39427143e76768fe176e8b9b -- ansi
verifying github.com/charmbracelet/x/[email protected]: checksum mismatch

❯ go list -m -json github.com/charmbracelet/x/[email protected]
{
        "Path": "github.com/charmbracelet/x/ansi",
        "Version": "v0.10.1",
        "Time": "2025-08-05T14:12:17Z",
        "GoMod": "/home/wold9168/go/pkg/mod/cache/download/github.com/charmbracelet/x/ansi/@v/v0.10.1.mod",
        "GoVersion": "1.23.0",
        "Sum": "h1:rL3Koar5XvX0pHGfovN03f5cxLbCF2YvLeyz7D2jVDQ=",
        "GoModSum": "h1:3RQDQ6lDnROptfpWuUVIUG64bD2g2BgntdxH0Ya5TeE=",
        "Origin": {
                "VCS": "git",
                "URL": "https://github.com/charmbracelet/x",
                "Subdir": "ansi",
                "Hash": "38fb69db254fb15a39427143e76768fe176e8b9b",
                "Ref": "refs/tags/ansi/v0.10.1"
        }
}

I've removed the entire $GOPATH. The problem still exists.

wold9168 avatar Nov 25 '25 20:11 wold9168

Hey there, I noticed something strange.

On my local machine, the package [email protected] downloads correctly and matches the expected checksum. However, inside a clean Docker environment running on the same physical machine, the downloaded package produces a different checksum—one that matches the one reported by @LeeBrotherston.

Since both environments are on the same host and share the same network, this issue doesn’t seem to be network-related. It’s likely due to differences in Go module caching, proxy settings, or how the module is resolved in each environment.

Problem machine(Go 1.24.9 on fedora42):

❯ GOPROXY=direct go mod download -json github.com/charmbracelet/x/[email protected]
{
        "Path": "github.com/charmbracelet/x/ansi",
        "Version": "v0.11.1",
        "Info": "/home/wold9168/go/pkg/mod/cache/download/github.com/charmbracelet/x/ansi/@v/v0.11.1.info",
        "GoMod": "/home/wold9168/go/pkg/mod/cache/download/github.com/charmbracelet/x/ansi/@v/v0.11.1.mod",
        "Zip": "/home/wold9168/go/pkg/mod/cache/download/github.com/charmbracelet/x/ansi/@v/v0.11.1.zip",
        "Dir": "/home/wold9168/go/pkg/mod/github.com/charmbracelet/x/[email protected]",
        "Sum": "h1:OSaQLe/B39MuYa0QwsJXJp378jUzFTqxMY99Zlz5U88=",
        "GoModSum": "h1:M49wjzpIujwPceJ+t5w3qh2i87+HRtHohgb5iTyepL0=",
        "Origin": {
                "VCS": "git",
                "URL": "https://github.com/charmbracelet/x",
                "Subdir": "ansi",
                "Hash": "09e6ba8ffbfcbc0f024403931f13531d3b993083",
                "Ref": "refs/tags/ansi/v0.11.1"
        }
}

Docker(Go 1.24.10 on fedora42):

[root@1064208702e8 [email protected]]# GOPROXY=direct go mod download -json github.com/charmbracelet/x/[email protected]
{
        "Path": "github.com/charmbracelet/x/ansi",
        "Version": "v0.11.1",
        "Info": "/root/go/pkg/mod/cache/download/github.com/charmbracelet/x/ansi/@v/v0.11.1.info",
        "GoMod": "/root/go/pkg/mod/cache/download/github.com/charmbracelet/x/ansi/@v/v0.11.1.mod",
        "Zip": "/root/go/pkg/mod/cache/download/github.com/charmbracelet/x/ansi/@v/v0.11.1.zip",
        "Dir": "/root/go/pkg/mod/github.com/charmbracelet/x/[email protected]",
        "Sum": "h1:iXAC8SyMQDJgtcz9Jnw+HU8WMEctHzoTAETIeA3JXMk=",
        "GoModSum": "h1:M49wjzpIujwPceJ+t5w3qh2i87+HRtHohgb5iTyepL0=",
        "Origin": {
                "VCS": "git",
                "URL": "https://github.com/charmbracelet/x",
                "Subdir": "ansi",
                "Hash": "09e6ba8ffbfcbc0f024403931f13531d3b993083",
                "Ref": "refs/tags/ansi/v0.11.1"
        }
}

I will now try to upgrade the Go toolchain version on the problematic machine to 1.24.10 to see what happens.

The problem remains.

wold9168 avatar Nov 25 '25 20:11 wold9168

I found something really interesting!

It seems that the clean env's git download a broken x/ansi(I use v0.10.3 for test) pkg.

See this screenshot:

Image

The left zip is from my problem machine and the right one is from the docker.

The only difference between is the JigokudaniMonkeyPark.png. The version in the Docker-downloaded ZIP appears to be corrupted.

I think this problem is related to git-lfs. And I am correct. There is a difference in the checksum of x/ansi between systems with and without Git LFS deployed. This difference is caused by the file JigokudaniMonkeyPark.png being stored in Git LFS.

@LeeBrotherston Please check if git-lfs is installed in your env. If so, I think the reason of this issue is that the go.sum is generated from an env without git-lfs.

See more:

  • https://stackoverflow.com/a/79662144/27058902
  • https://github.com/thavaahariharangit/go_checksum_mismatch
  • https://github.com/golang/go/issues/41708

In summary, when building any package that depends on files containing LFS, if a checksum mismatch error occurs, please first add GIT_LFS_SKIP_SMUDGE=1 to ensure that the locally generated checksum matches the record in go.sum.

btw, it is worth mentioning that if you use the GOPROXY command like GOPROXY=https://goproxy.cn,direct go mod download to download Go packages, the packages you download will not have this mismatch issue.

wold9168 avatar Nov 25 '25 21:11 wold9168

@wold9168 oh yes, that appears to be it!

For what it's worth I don't think the file is corrupted, I think that when you don't have git-lfs installed you get a smaller file that's a pointer to the real file. So presumably that pointer file is being checksum'd instead of the real file.

That wild because that means that although we're having the issue due to using git-lfs, we actually have the right checksum and sumdb, etc have the wrong checksum.

While I'll use GIT_LFS_SKIP_SMUDGE=1 to get around this, I suspect that the real answer is not to host that file using git-lfs in the first place so that everyone (including sumdb) gets the right signature.

LeeBrotherston avatar Nov 26 '25 14:11 LeeBrotherston

ah, now this all makes sense!

I think this might fix it, then: https://github.com/charmbracelet/x/pull/661

caarlos0 avatar Nov 26 '25 14:11 caarlos0

just pushed ansi/v0.11.2 - feel free to test it and let me know if it fixes the issue 🙏🏻

caarlos0 avatar Nov 26 '25 16:11 caarlos0

and thanks everyone for helping figuring this out 🙏🏻

caarlos0 avatar Nov 26 '25 16:11 caarlos0