Upgrade to v18.4 failed, cannot install both kubelet-1.18.4-1.x86_64 and kubelet-1.18.4-0.x86_64

[StefanSa]

What happened: on the master centos-8 there is the following error message when trying to update the packages.

Problem: cannot install both kubelet-1.18.4-1.x86_64 and kubelet-1.18.4-0.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.7.5-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.3-0.x86_64

We don't see this problem in the nodes centos-7 Likewise, we did not see this error in the master centtos-8 in the last update to v18.3.

How to reproduce it (as minimally and precisely as possible): yum update / dnf update

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: vmware
• OS (e.g: cat /etc/os-release): centos-8

[StefanSa]

/sig Release

[StefanSa]

This helps me as a short-term solution. yum update --skip-broken --nobest

but does not fix the actual packet conflict.

yum update
 Problem: cannot install both kubelet-1.18.4-0.x86_64 and kubelet-1.18.4-1.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.8.6-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.4-1.x86_64

[justaugustus]

@StefanSa -- What happens when you completely remove kubelet and kubernetes-cni and then attempt to install one of the following kubelets:

• kubelet-1.18.5-0
• kubelet-1.18.4-1

(Related to https://github.com/kubernetes/kubernetes/issues/92242.) /assign /area release-eng cc: @kubernetes/release-engineering

[StefanSa]

@justaugustus i remove kubelet and kubernetes-cniand installed it again successfully with: yum install kubelet-1.18.5 kubeadm-1.18.5, however, a yum update still fails.

yum update
Problem: cannot install both kubelet-1.18.4-0.x86_64 and kubelet-1.18.5-0.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.8.6-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.5-0.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken'
to skip uninstallable packages or '--nobest' to use not only best candidate packages)

As already mentioned, we don't see that on our nodes that are based on centos 7, currently we only see this problem with the master that is based on centos 8.

[mrbobbytables]

There does still seem to be an issue on centos8 I had the same issue when testing with 1.17.8, 1.18.1, 1.18.3, and 1.18.5

root@a77d16cf886f /]# yum info kubelet
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:06:28 ago on Thu Jul 30 12:22:37 2020.
Installed Packages
Name         : kubelet
Version      : 1.18.5
Release      : 0
Architecture : x86_64
Size         : 108 M
Source       : kubelet-1.18.5-0.src.rpm
Repository   : @System
From repo    : kubernetes
Summary      : Container cluster management
URL          : https://kubernetes.io
License      : ASL 2.0
Description  : The node agent of Kubernetes, the container cluster manager.

Available Packages
Name         : kubelet
Version      : 1.18.6
Release      : 0
Architecture : x86_64
Size         : 21 M
Source       : kubelet-1.18.6-0.src.rpm
Repository   : kubernetes
Summary      : Container cluster management
URL          : https://kubernetes.io
License      : ASL 2.0
Description  : The node agent of Kubernetes, the container cluster manager.

[root@a77d16cf886f /]# yum update
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:06:33 ago on Thu Jul 30 12:22:37 2020.
Error:
 Problem: cannot install both kubelet-1.18.6-0.x86_64 and kubelet-1.18.4-0.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.8.6-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.5-0.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

[huataihuang]

I still had same issue with 1.18.6 upgade to 1.19.0.

I use centos 8.

#dnf update
...
Error:
 Problem: cannot install both kubelet-1.19.0-0.x86_64 and kubelet-1.18.4-0.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.8.6-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.6-0.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

#dnf info kubelet
Last metadata expiration check: 0:39:13 ago on Wed 02 Sep 2020 09:18:41 AM CST.
Installed Packages
Name         : kubelet
Version      : 1.18.6
Release      : 0
Architecture : x86_64
Size         : 108 M
Source       : kubelet-1.18.6-0.src.rpm
Repository   : @System
From repo    : kubernetes
Summary      : Container cluster management
URL          : https://kubernetes.io
License      : ASL 2.0
Description  : The node agent of Kubernetes, the container cluster manager.

Available Packages
Name         : kubelet
Version      : 1.19.0
Release      : 0
Architecture : x86_64
Size         : 19 M
Source       : kubelet-1.19.0-0.src.rpm
Repository   : kubernetes
Summary      : Container cluster management
URL          : https://kubernetes.io
License      : ASL 2.0
Description  : The node agent of Kubernetes, the container cluster manager.

[huataihuang]

I add follow config to /etc/yum.conf to exclude conflict kubelet

exclude=kubelet-1.18.4-0 kubelet-1.17.7-0 kubelet-1.16.11-0

then use dnf upgrade to upgrade kubelet 1.19.0 without problem.

I hope it can help you.

[laundry-96]

@huataihuang What repository url do you have? I've been using https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 (which is what the kubernetes website tells me to use) and it doesn't seem to have kubelet-1.19 in it...

[huataihuang]

@huataihuang What repository url do you have? I've been using https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 (which is what the kubernetes website tells me to use) and it doesn't seem to have kubelet-1.19 in it...

https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

here is my /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

I was follow kubernetes website guide to deploy, and I add exclude config by my experience, maybe you not meet this version conflict.

[justaugustus]

/assign @saschagrunert /unassign

Sascha -- can you take a look or assign to someone on @kubernetes/release-engineering to investigate?

The following issues are similar classes of issues for apt/yum:

• https://github.com/kubernetes/kubernetes/issues/92698
• https://github.com/kubernetes/kubernetes/issues/92587
• https://github.com/kubernetes/kubernetes/issues/92586
• https://github.com/kubernetes/release/issues/3044

[saschagrunert]

/help Not sure if we can solve this without re-building those packages.

[k8s-ci-robot]

@saschagrunert: This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to this:

/help Not sure if we can solve this without re-building those packages.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Mikhail2577]

Hi everyone, seems like I can't install kubeadm or kubelet on my fresh install of CentOS7.

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

Loaded plugins: fastestmirror, product-id, search-disabled-repos, subscription-manager

This system is not registered with an entitlement server. You can use subscription-manager to register.

Loading mirror speeds from cached hostfile

• base: ftp.belnet.be
• extras: ftp.belnet.be
• updates: ftp.belnet.be No package kubelet available. No package kubeadm available. No package kubectl available. Error: Nothing to do

[cyshallchan]

I use centos 8, same issue.

[likan999]

Just curious, why doesn't Kubernetes support CentOS 8? Seems we still need to use el7 repos on CentOS 8 after it has been released for more than a year.

[yabb85]

I use centos 8, same issue.

[boogiefromzk]

Same on centos 8

[SnailDove]

there also is the same issue on my CentOS 8 system

[boogiefromzk]

Still actual even for v 1.21

[cpanato]

This happened for 1.18 right? this is not supposed to occur on 1.19+ releases, I guess.

1.18 reach the end of life and we are not supporting this version anymore. I guess we can close this issue.

comments? @saschagrunert @justaugustus @hasheddan

[vrusinov]

Conflicts similar to above still happen on CentOS 8 with 1.21 rpms installed. exclude= in yum.conf mitigates it, but it would be nice to clean this up somehow.

[TheFrisianClause]

Well I am trying to automate this process with Ansible as well on RockyLinux 8 but I seem to get this error to. This is the debug:

fatal: [kube-worker-02]: FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \n Problem: cannot install both kubelet-1.18.4-0.x86_64 and kubelet-1.22.0-0.x86_64\n - cannot install the best update candidate for package kubernetes-cni-0.8.7-0.x86_64\n - cannot install the best update candidate for package kubelet-1.22.0-0.x86_64", "rc": 1, "results": []}

I also tried 'Allowerasing: yes' but this seem not to resolve the problem unfortunately...

I have the following kubernetes repo configured (according to the kubernetes documentation)

[kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg exclude = kubelet kubeadm kubectl

Is there a reason why this happens and if so is there some kind of fix on the way to resolve this? As it seems to be going on for quite a while now.

[Michael-M-Zhang]

Same issue on my centos system. sudo yum update Repository extras is listed more than once in the configuration Last metadata expiration check: 2:48:25 ago on Wed 15 Sep 2021 07:04:01 PM CST. Error: Problem: cannot install both kubelet-1.18.4-0.x86_64 and kubelet-1.22.1-0.x86_64

• cannot install the best update candidate for package kubernetes-cni-0.8.7-0.x86_64
• cannot install the best update candidate for package kubelet-1.22.1-0.x86_64 (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[vrusinov]

/remove-lifecycle stale

This is still happening. In general kubernetes RPM repository is not in the best shape.

[Uj947nXmRqV2nRaWshKtHzTvckUUpD]

still heppening: Problem: cannot install both kubelet-1.18.4-0.x86_64 and kubelet-1.23.2-0.x86_64

• cannot install the best update candidate for package kubernetes-cni-0.8.7-0.x86_64
• cannot install the best update candidate for package kubelet-1.23.2-0.x86_64

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ViliusS]

/remove-lifecycle stale

[ViliusS]

So almost 2 years later and dozen of duplicate issues filled against this bug, we still have the same issue :(

[Uj947nXmRqV2nRaWshKtHzTvckUUpD]

I add follow config to /etc/yum.conf to exclude conflict kubelet

exclude=kubelet-1.18.4-0 kubelet-1.17.7-0 kubelet-1.16.11-0

then use dnf upgrade to upgrade kubelet 1.19.0 without problem.

I hope it can help you.

This is the workaround. I'll repost this, so that it is not lost after 2 years of comments. At least to help others who bump into this issue..