Failed to install ingress-nginx plugin with krew!

[zhumeishuai]

Installing ingress-nginx plugin with krew failed with an error download https://github.com/kubernetes/ingress-nginx/releases/download/controller-0.31.0/kubectl-ingress_nginx-linux-amd64.tar.gz 404. I tried my best on the search engines, but couldn't find any relevant information.

[zhumeishuai]

Is there any way to solve it?

[zhumeishuai]

@strongjz Is there any way to solve it?

[strongjz]

0.31.0 controller is no longer supported. To my understanding, the plugin version should match the controller version.

[gianarb]

Hello @strongjz I have facing the same issue.

I use helm to install nginx in my kubernetes cluster. Currently this is the helm version:

  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  version    = "4.11.3"

At the best of my knowledge it is the latest released by this project.

Looking at the nginx controller image deployed this is what I have:

  Containers:
   controller:
    Image:           registry.k8s.io/ingress-nginx/controller:v1.11.3@sha256:d56f135b6462cfc476447cfe564b83a45e8bb7da2774963b00d12161112270b7
    Ports:           80/TCP, 443/TCP, 8443/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault

I can't find 0.31.0 anywhere in my system, so I am not sure why krew is looking for such version. Unfortunately I am debugging an issue for one of my cluster. I can't really dig deeper, but I will be back with more if I can find what is going on.

In the meantime I am gonna build the plugin from source

[mhbahmani]

Same issue here. Is there any solution for this?

[threadcrux]

See my related issue filed here in krew repo: https://github.com/kubernetes-sigs/krew-index/issues/4254

This is not a solution but shedding more light on the situation. The plugin definition has not been updated in 4+ years.

There is also a secondary issue, the plugin is based upon the controller releases that used to provide builds for different platforms/architectures. It no longer looks like the release system is providing builds.

Ping: @cpanato @Gacko @puerco @rikatz @strongjz @tao12345666333

Sorry about the ping blast. What needs to be done to get platform/architecture builds reestablished for the above? I can provide the fix in the krew repo but there is nothing to point the plugin definitions at.

I am also unsure of which version would be best for the plugin definition

[threadcrux]

I started to reverse engineer what happened. The MakeFile no longer has a build-plugin target and the associated build-plugin.sh script is missing.

I filed an issue for those here: https://github.com/kubernetes/ingress-nginx/issues/12436

Looks like it was removed here, and then nothing has happened since: https://github.com/kubernetes/ingress-nginx/pull/8812

Not sure why removing the build script for manual building was the idea thrown around. Anyways there is a github action for this but not entirely sure its working

[strongjz]

@threadcrux if you want to help ensure this doesnt happen again, you could add some ci to test the krew build on PR.

[threadcrux]

@threadcrux if you want to help ensure this doesnt happen again, you could add some ci to test the krew build on PR.

I can start to think about it. I have a little bit of over commitment at the moment.

Questions:

• This would test the plugin after build -separate of krew plugin? Or post krew install? Or both?
• Are there any docs or previous implementations of testing in this area? How do you see this occurring from a high level standpoint. I'd like to see anything that's preexisting

It's definitely a good idea even if the initial implementation is super simple (i.e. testing a single piece of functionality to start). This would validate the releases are being made and it's usable from a deployment standpoint, not necessarily a functionality standpoint, but its a move in the right direction.

EDIT: Missed this phrasing

test the krew build on PR

Gotcha I see what you're thinking, add the test on the other side of that repo

[strongjz]

/kind bug /priority backlog /triage accepted /assign @strongjz

[strongjz]

issues with our release for 1.12 have been resolved, we are looking to release 1.12.0, next patch for 1.11 and the final for 1.10 soon.

We should know then if the patch in the GHA for https://github.com/kubernetes/ingress-nginx/pull/12439 fixed the krew issue.

[allanhung]

From the GHA logs, it seems that the goreleaser configuration file (.goreleaser.yaml) needs to include version: 2.

version: 2
project_name: ingress-nginx
release:
  github:
    owner: kubernetes
...

[olix0r]

Is there a relatively straightforward way to fetch or build the binary manually? The docs only mention krew, afaict.

Barring that, is there a way to perform kubectl ingress-nginx backends manually (e.g. with port forwards?)

[olix0r]

I was able to work this out with something like:

:; go install k8s.io/ingress-nginx/cmd/[email protected]
...
:; go/bin/plugin
A kubectl plugin for inspecting your ingress-nginx deployments

Usage:
  ingress-nginx [command]
  ...

[zektorum]

I was able to work this out with something like: :; go install k8s.io/ingress-nginx/cmd/[email protected]

After your command you should to run these commands

cp go/bin/plugin .krew/bin/kubectl-ingress_nginx
mkdir -p .krew/store/ingress-nginx/v1.11.4
ln -s /home/$USER/.krew/bin/kubectl-ingress_nginx /home/$USER/.krew/store/ingress-nginx/v1.11.4/ingress-nginx

and after that kubectl ingress-nginx will work

[allanhung]

If you want to build it yourself, you can try the following script: (it works for my mac)

brew install goreleaser gsed
git clone https://github.com/kubernetes/ingress-nginx
cd ingress-nginx
gsed -i -e '1i\version: 2' -e "/project_name: ingress-nginx/a\snapshot:\n  version_template: \"{{ .Tag }}-dirty\"" .goreleaser.yaml
gsed -i -e "/linux/d" -e "/windows/d" -e "/amd64/d" .goreleaser.yaml
goreleaser release --snapshot --clean
sudo cp $PWD/dist/ingress-nginx_darwin_arm64_v8.0/kubectl-ingress-nginx /usr/local/bin/kubectl-ingress_nginx

[manios]

Hi everyone!

I have used the information provided by @zektorum and @allanhung to build the plugin locally and it worked! Thanks! I was curious about the whole plugin system and indices so I have created a custom index which you can use until this issue is resolved. You can find it here: manios/krew-ingress-nginx. The manual build steps that I followed are here: build.md

In order to install the plugin through this custom index , first of all you have to add this custom index:

kubectl krew index add ingress-manios https://github.com/manios/krew-ingress-nginx.git

And then install the plugin:

kubectl krew install ingress-manios/ingress-nginx-cm 

Then you can use it with kubectl ingress-nginx-cm.

Thank you for your valuable information!

[vrubiolo]

Hello there! I hit the bug today while installing the nginx-ingress controller in my cluster wanting to try the krew plugin. @allanhung : I tried to use your instructions for rebuilding but this fails at the release step:

$ git clone https://github.com/kubernetes/ingress-nginx
Cloning into 'ingress-nginx'...
remote: Enumerating objects: 130154, done.
remote: Counting objects: 100% (579/579), done.
remote: Compressing objects: 100% (212/212), done.
remote: Total 130154 (delta 501), reused 369 (delta 366), pack-reused 129575 (from 4)
Receiving objects: 100% (130154/130154), 131.98 MiB | 34.58 MiB/s, done.
Resolving deltas: 100% (75633/75633), done.
$ cd ingress-nginx
$ gsed -i -e '1i\version: 2' -e "/project_name: ingress-nginx/a\snapshot:\n  version_template: \"{{ .Tag }}-dirty\"" .goreleaser.yaml
$ gsed -i -e "/linux/d" -e "/windows/d" -e "/amd64/d" .goreleaser.yaml
$ goreleaser release --snapshot --clean
  • skipping announce, publish and validate...
  • cleaning distribution directory
  • loading environment variables
  • getting and validating git state
    • git state                                      commit=97bbec446b6fc24129846c9b602681e239ea2503 branch=main current_tag=controller-v1.11.0 previous_tag=controller-v1.10.1 dirty=true
    • pipe skipped                                   reason=disabled during snapshot mode
  • parsing tag
  ⨯ release failed after 0s                  error=failed to parse tag 'controller-v1.11.0' as semver: Invalid Semantic Version

Could it be because of a change in goreleaser and how it handles version tags (expecting a real number and not a prefix like controller-? Mine is:

$ goreleaser --version
  ____       ____      _
 / ___| ___ |  _ \ ___| | ___  __ _ ___  ___ _ __
| |  _ / _ \| |_) / _ \ |/ _ \/ _` / __|/ _ \ '__|
| |_| | (_) |  _ <  __/ |  __/ (_| \__ \  __/ |
 \____|\___/|_| \_\___|_|\___|\__,_|___/\___|_|
goreleaser: Release engineering, simplified.
https://goreleaser.com

GitVersion:    2.6.1
GitCommit:     b6bb03ddc28d6de71a37012107cda26af53cc116
GitTreeState:  clean
BuildDate:     2025-01-22T11:11:07
BuiltBy:       homebrew
GoVersion:     go1.23.5
Compiler:      gc
ModuleSum:     unknown
Platform:      darwin/arm64

Thanks for your assistance !

[vrubiolo]

Ok so it looks like goreleaser enforces semantic versioning on tags even in local release mode. This can be overriden via the environment.

Thus, the release command can be: GORELEASER_CURRENT_TAG=v1.11.0 goreleaser release --snapshot --clean

This allows the build to succeed on my end.

PS: it also looks like the format: directive needs to be updated to formats: ['tar.gz']

but this is not blocking the build, only goreleaser check.

[vrubiolo]

Ok so to complete the reproduceability steps, once you have rebuilt the plugin using the abovementioned instructions, you can install it locally (as per the krew developer notes for testing):

To do so:

1. fetch the archive you just built from under ingress-nginx/dist/kubectl-ingress-nginx_linux_amd64.tar.gz
2. get ingress-nginx.yaml from below, replacing the sha256: attribute with your archive associated checksum
3. install the plugin using kubectl krew install --manifest ingress-nginx.yaml --archive kubectl-ingress-nginx_linux_amd64.tar.gz

You can then use the plugin as normal:

$ k krew install --manifest krew/ingress-nginx.yaml --archive krew/kubectl-ingress-nginx_linux_amd64.tar.gz
Installing plugin: ingress-nginx
Installed plugin: ingress-nginx
\
 | Use this plugin:
 |      kubectl ingress-nginx
 | Documentation:
 |      none
/
$ k ingress-nginx -h
A kubectl plugin for inspecting your ingress-nginx deployments

Usage:
  ingress-nginx [command]

Available Commands:
  backends    Inspect the dynamic backend information of an ingress-nginx instance
  certs       Output the certificate data stored in an ingress-nginx pod
  completion  Generate the autocompletion script for the specified shell
[...]

ingress-nginx.yaml:

apiVersion: krew.googlecontainertools.github.com/v1alpha2
kind: Plugin
metadata:
  name: ingress-nginx
spec:
  version: v0.0.1
  platforms:
    - bin: ./kubectl-ingress-nginx
      uri: file://kubectl-ingress-nginx_linux_amd64.tar.gz
      sha256: a34c20550c26d3f776f0f97782f1ae56f4d75008a3ac69e77df3884036336ca0
      selector:
        matchLabels:
          os: linux
          arch: amd64
      files:
      - from: kubectl-ingress-nginx
        to: kubectl-ingress-nginx
  shortDescription: Custom rebuild of the ingress-nginx krew plugin
  homepage: none
  description: |
    Unofficial, locally rebuilt version of the ingress-nginx plugin

Alternatively, you can also use @manios repository (although I am not sure where the binaries are coming from).

[github-actions[bot]]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.

[jcouturest]

So, this is still an issue. Just because we can build and install locally doesn't mean we should be, can someone fix the GHA job please? https://github.com/kubernetes/ingress-nginx/actions/runs/14049108605

[jcouturest]

Trying to migrate because of the CVE, and can't use this to get the actual hosts its trying to use. Either that, or just remove krew from the docs, and document how to get the actual upstream hosts. The last successful krew release was 5 years ago https://github.com/kubernetes-sigs/krew-index/blob/master/plugins/ingress-nginx.yaml

[elopsod]

works for me

dockerfile

FROM golang:1.24.2-alpine3.21
RUN go install k8s.io/ingress-nginx/cmd/[email protected]

shell

docker build -t controller:0.0.0 -f Dockerfile .
docker run --name tmp -d controller:0.0.0 sleep infinity
docker cp tmp:/go/bin/plugin /tmp/kubectl-ingress_nginx
docker rm --force tmp
mv /tmp/kubectl-ingress_nginx /usr/local/bin/

[igoratencompass]

The process from @zektorum worked for me for the latest controller version 1.12.1 but the plugin itself does not work:

$ kubectl ingress-nginx conf -n ingress-nginx
exit status 1

$ kubectl ingress-nginx backends -n ingress-nginx
exit status 1

Kubernetes version is 1.31

[DDDFiish]

any updates on this?

[btreecat]

Still having issues trying to install via krew

kubectl krew install ingress-nginx
Updated the local copy of plugin index.
Installing plugin: ingress-nginx
W0618 15:36:44.346284   58772 install.go:164] failed to install plugin "ingress-nginx": install failed: failed to unpack into staging dir: failed to unpack the plugin archive: failed to obtain plugin archive: failed to download "https://github.com/kubernetes/ingress-nginx/releases/download/controller-0.31.0/kubectl-ingress_nginx-linux-amd64.tar.gz", status code 404
failed to install some plugins: [ingress-nginx]: install failed: failed to unpack into staging dir: failed to unpack the plugin archive: failed to obtain plugin archive: failed to download "https://github.com/kubernetes/ingress-nginx/releases/download/controller-0.31.0/kubectl-ingress_nginx-linux-amd64.tar.gz", status code 404

[spkane]

I couldn't install it with krew, but this worked fine on an M4 (arm64) Mac:

git clone [email protected]:kubernetes/ingress-nginx.git
cd ingress-nginx/cmd/plugin
go build .
mv plugin /usr/local/bin/kubectl-ingress_nginx
kubectl ingress_nginx help

[nogweii]

There wouldn't be any progress until the associated PR (#13157) is merged in; alternatively you can manually download and build the plugin as @spkane demonstrates.