vllm-project
Bug: Build fails with "invalid array length -delta * delta" due to golang.org/x/tools version incompatibility
What happened:
When running make test or make generate, the build fails with the following error:
# golang.org/x/tools/internal/tokeninternal
../../../../pkg/mod/golang.org/x/[email protected]/internal/tokeninternal/tokeninternal.go:64:9: invalid array length -delta * delta (constant -256 of type int64)
This occurs during the download or execution of controller-gen and code-generator, likely due to an incompatibility between the Go version and a recent breaking change in golang.org/x/tools.
Environment:
- Go version: (e.g., go version go1.25.1 darwin/arm64)
- OS: mac
Steps to Reproduce:
make generate # or make test
@googs1025 I could take this :-)
@googs1025 @Jeffwan fix will result into changing go version from 1.22.x to 1.24.x / 1.25.x for AIBrix project, would that be fine ?
Ideally, Go provides support for last two released version (like applying security fix etc), so it's better to upgrade to go1.24 / go1.25.
FYI -- we have two go.mod file so this change will be apply to both places. (link1, link2)
Plz confirm, after that I will start working on this 👍
Is it possible to avoid the upgrade?
Let me check details and get back here 👍
@googs1025 Investigation so far ..
Step-1: I have started from scratch; freshly git clone aibrix repo and then run make generate
make generate gives below error,
$ aibrix git:(main) make generate
mkdir -p /home/nurali/go/src/github.com/nurali-techie/aibrix/bin
Downloading sigs.k8s.io/controller-tools/cmd/[email protected]
# golang.org/x/tools/internal/tokeninternal
../../../../pkg/mod/golang.org/x/[email protected]/internal/tokeninternal/tokeninternal.go:64:9: invalid array length -delta * delta (constant -256 of type int64)
make: *** [Makefile:485: /home/nurali/go/src/github.com/nurali-techie/aibrix/bin/controller-gen-v0.16.1] Error 1
Above is Issue-1.
Basically [email protected] uses golang.org/x/tools v0.24.0 (check code_link) which fails when run internal/tokeninternal/tokeninternal.go:64:9 (check code_link)
Fix for Issue-1:
Use controller-tools v0.16.4 instead of v0.16.1 (changes in aibrix code_link)
controller-tools v0.16.4 uses golang.org/x/tools v0.26.0 which doesn't have Issue-1.
Step-2: After above fix, again run make generate
make generate gives below error,
$ aibrix git:(main) ✗ make generate
/home/nurali/go/src/github.com/nurali-techie/aibrix/bin/controller-gen-v0.16.4 object:headerFile="hack/boilerplate.go.txt" paths="./..."
./hack/update-codegen.sh go /bin
>> Using /home/nurali/go/pkg/mod/k8s.io/[email protected]
# golang.org/x/tools/internal/tokeninternal
../../golang.org/x/[email protected]/internal/tokeninternal/tokeninternal.go:64:9: invalid array length -delta * delta (constant -256 of type int64)
make: *** [Makefile:113: generate] Error 1
Above is Issue-2.
Basically, [email protected] uses golang.org/x/[email protected] (check code_link) and this x/tools gives same error invalid array length
Fix for Issue-2:
I checked all patched version [email protected] (till v0.31.14) but it has same issue. Issue is fixed by changing minor version v0.32.0 in aibrix go.mod file (here code_link)
To fix, I ran go get k8s.io/[email protected] and that also changed go version to go 1.23.0 in go.mod file.
After that, again run make generate which failed with below error,
$ aibrix git:(main) ✗ make generate
/home/nurali/go/src/github.com/nurali-techie/aibrix/bin/controller-gen-v0.16.4 object:headerFile="hack/boilerplate.go.txt" paths="./..."
../../../../pkg/mod/k8s.io/[email protected]/pkg/util/httpstream/spdy/connection.go:25:2: missing go.sum entry for module providing package github.com/moby/spdystream (imported by k8s.io/apimachinery/pkg/util/httpstream/spdy); to add:
go get k8s.io/apimachinery/pkg/util/httpstream/[email protected]
test/utils/utils.go:25:2: missing go.sum entry for module providing package github.com/onsi/ginkgo/v2 (imported by github.com/vllm-project/aibrix/test/utils); to add:
go get github.com/vllm-project/aibrix/test/utils
Error: not all generators ran successfully
run `controller-gen object:headerFile=hack/boilerplate.go.txt paths=./... -w` to see all available markers, or `controller-gen object:headerFile=hack/boilerplate.go.txt paths=./... -h` for usage
make: *** [Makefile:112: generate] Error 1
Overall, to fix Issue-2, we need to upgrade to go 1.23.x (from go 1.22.x). So far, I don't see we can avoid go version upgrade.
@googs1025 need your input/thoughts on above; in case I am missing something here, Thanks.
Can't we solve this problem by upgrading controller-gen or code-generator?
Can't we solve this problem by upgrading controller-gen or code-generator?
@googs1025 I don't think so.
Just to unblock make generate; I hard-coded code-generator version to v0.32.0 here (check code_link) instead of grep from go.mod file.
It passed make generate but failed make test with below error,
Generating applyconfig code for 3 targets
Generating client code for 3 targets
Generating lister code for 3 targets
Generating informer code for 3 targets
go fmt ./...
go vet ./...
# github.com/vllm-project/aibrix/pkg/client/clientset/versioned/typed/autoscaling/v1alpha1
pkg/client/clientset/versioned/typed/autoscaling/v1alpha1/autoscaling_client.go:90:37: undefined: rest.CodecFactoryForGeneratedClient
# github.com/vllm-project/aibrix/pkg/client/clientset/versioned/typed/model/v1alpha1
pkg/client/clientset/versioned/typed/model/v1alpha1/model_client.go:90:37: undefined: rest.CodecFactoryForGeneratedClient
# github.com/vllm-project/aibrix/pkg/client/clientset/versioned/typed/orchestration/v1alpha1
pkg/client/clientset/versioned/typed/orchestration/v1alpha1/orchestration_client.go:100:37: undefined: rest.CodecFactoryForGeneratedClient
To fix above issue, I need to upgrade k8s.io/client-go version from v0.31.8 to v0.32.0 (none of patched version of k8s.io/client-go worked; so need to upgrade minor version).
With above fix, it changed go version to go 1.23.0 in go.mod file.
Overall, we need to upgrade from v0.31.8 to v0.32.0 (for k8s.io/xxx packages) which in turns upgrade go version to go 1.23.x
Let me raise PR which allow to run make test using go 1.25.x
Note: I am using linux machine with go version go1.25.4 linux/amd64.
@googs1025 here you go; I have raised PR (link), check go.mod file changes with go version change to 1.23 (check -- code_link)
This is the minimum changes needed to run make test successfully using go 1.25.x (I have used go1.25.4 linux/amd64).
I will wait for feedback before resuming work on this 👍
@googs1025 overall either we need to upgrade go version or we need to stay with limitation that we can't use go1.25 for building aibrix.
Let me know if you want anything here or we can close this issue with above limitation 🙏