ceph-csi icon indicating copy to clipboard operation
ceph-csi copied to clipboard

Address security vulnarabilities in the image/binary/repo

Open humblec opened this issue 2 years ago • 24 comments

Describe the bug

We are getting many reports against Ceph CSI image and the vulnerabilities it hold. it is required/better to address as much as we can. as part of this effort I have started enabling trvivy scanner in the repo via https://github.com/ceph/ceph-csi/pull/3537 and initial report says

quay.io/cephcsi/cephcsi:test (redhat 8.6)
=========================================
Total: 14 (UNKNOWN: 0, LOW: 1, MEDIUM: 12, HIGH: 1, CRITICAL: 0)

┌───────────────────────┬────────────────┬──────────┬──────────────────────────────────────┬────────────────────────────────────────┬──────────────────────────────────────────────────────────────┐
│        Library        │ Vulnerability  │ Severity │          Installed Version           │             Fixed Version              │                            Title                             │
├───────────────────────┼────────────────┼──────────┼──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ gnutls                │ CVE-2022-2509  │ MEDIUM   │ 3.6.16-5.el8                         │ 3.6.16-5.el8_6                         │ gnutls: Double free during gnutls_pkcs7_verify               │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-2509                    │
├───────────────────────┼────────────────┼──────────┼──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ libksba               │ CVE-2022-3515  │ HIGH     │ 1.3.5-7.el8                          │ 1.3.5-8.el8_6                          │ libksba: integer overflow may lead to remote code execution  │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-3515                    │
├───────────────────────┼────────────────┼──────────┼──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ platform-python       │ CVE-2015-20107 │ MEDIUM   │ 3.6.8-47.el8                         │ 3.6.8-47.el8_6                         │ python: mailcap: findmatch() function does not sanitize the  │
│                       │                │          │                                      │                                        │ second argument                                              │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2015-20107                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2022-0391  │          │                                      │                                        │ python: urllib.parse does not sanitize URLs containing ASCII │
│                       │                │          │                                      │                                        │ newline and tabs                                             │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-0391                    │
├───────────────────────┼────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│ platform-python-devel │ CVE-2015-20107 │          │                                      │                                        │ python: mailcap: findmatch() function does not sanitize the  │
│                       │                │          │                                      │                                        │ second argument                                              │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2015-20107                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2022-0391  │          │                                      │                                        │ python: urllib.parse does not sanitize URLs containing ASCII │
│                       │                │          │                                      │                                        │ newline and tabs                                             │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-0391                    │
├───────────────────────┼────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│ python3-libs          │ CVE-2015-20107 │          │                                      │                                        │ python: mailcap: findmatch() function does not sanitize the  │
│                       │                │          │                                      │                                        │ second argument                                              │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2015-20107                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2022-0391  │          │                                      │                                        │ python: urllib.parse does not sanitize URLs containing ASCII │
│                       │                │          │                                      │                                        │ newline and tabs                                             │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2022-0391                    │
├───────────────────────┼────────────────┤          ├──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ python3-scipy         │ CVE-2021-20270 │          │ 1.0.0-21.module_el8.5.0+771+e5d9a225 │ 1.0.0-21.module+el8.5.0+10916+41bd434d │ python-pygments: Infinite loop in SML lexer may lead to DoS  │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2021-20270                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2021-27291 │          │                                      │                                        │ python-pygments: ReDoS in multiple lexers                    │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2021-27291                   │
├───────────────────────┼────────────────┤          ├──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ python36              │ CVE-2021-20270 │          │ 3.6.8-38.module_el8.5.0+895+a459eca8 │ 3.6.8-38.module+el8.5.0+12207+5c5719bc │ python-pygments: Infinite loop in SML lexer may lead to DoS  │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2021-20270                   │
│                       ├────────────────┤          │                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2021-27291 │          │                                      │                                        │ python-pygments: ReDoS in multiple lexers                    │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2021-27291                   │
├───────────────────────┼────────────────┤          ├──────────────────────────────────────┼────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ sqlite-libs           │ CVE-2020-35527 │          │ 3.26.0-16.el8                        │ 3.26.0-16.el8_6                        │ sqlite: Out of bounds access during table rename             │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2020-35527                   │
│                       ├────────────────┼──────────┤                                      │                                        ├──────────────────────────────────────────────────────────────┤
│                       │ CVE-2020-35525 │ LOW      │                                      │                                        │ sqlite: Null pointer derreference in src/select.c            │
│                       │                │          │                                      │                                        │ https://avd.aquasec.com/nvd/cve-2020-35525                   │
└───────────────────────┴────────────────┴──────────┴──────────────────────────────────────┴────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘

Python (python-pkg)
===================
Total: 3 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 1, CRITICAL: 2)

┌───────────────────────────────────────────────────┬────────────────┬──────────┬───────────────────┬───────────────┬──────────────────────────────────────────────────────────────┐
│                      Library                      │ Vulnerability  │ Severity │ Installed Version │ Fixed Version │                            Title                             │
├───────────────────────────────────────────────────┼────────────────┼──────────┼───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ joblib (PKG-INFO)                                 │ CVE-2022-21797 │ CRITICAL │ 0.16.0            │ 1.2.0         │ The package joblib from 0 and before 1.2.0 are vulnerable to │
│                                                   │                │          │                   │               │ Arbitrary...                                                 │
│                                                   │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2022-21797                   │
├───────────────────────────────────────────────────┼────────────────┤          ├───────────────────┼───────────────┼──────────────────────────────────────────────────────────────┤
│ scikit-learn (scikit_learn-0.19.1-py3.6.egg-info) │ CVE-2020-13092 │          │ 0.19.1            │ 0.23.1        │ ** DISPUTED ** scikit-learn (aka sklearn) through 0.23.0 can │
│                                                   │                │          │                   │               │ unseriali ...                                                │
│                                                   │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2020-13092                   │
│                                                   ├────────────────┼──────────┤                   ├───────────────┼──────────────────────────────────────────────────────────────┤
│                                                   │ CVE-2020-28975 │ HIGH     │                   │ 0.24.dev0     │ ** DISPUTED ** svm_predict_values in svm.cpp in Libsvm v324, │
│                                                   │                │          │                   │               │ as used in...                                                │
│                                                   │                │          │                   │               │ https://avd.aquasec.com/nvd/cve-2020-28975                   │
└───────────────────────────────────────────────────┴────────────────┴──────────┴───────────────────┴───────────────┴──────────────────────────────────────────────────────────────┘

usr/local/bin/cephcsi (gobinary)
================================
Total: 6 (UNKNOWN: 0, LOW: 0, MEDIUM: 4, HIGH: 1, CRITICAL: 1)

┌────────────────────────────┬────────────────┬──────────┬───────────────────┬────────────────────────────┬──────────────────────────────────────────────────────────────┐
│          Library           │ Vulnerability  │ Severity │ Installed Version │       Fixed Version        │                            Title                             │
├────────────────────────────┼────────────────┼──────────┼───────────────────┼────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ github.com/hashicorp/vault │ CVE-2020-16250 │ CRITICAL │ v1.4.2            │ 1.5.1, 1.5.1, 1.2.5, 1.3.8 │ Authentication Bypass by Spoofing                            │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2020-16250                   │
│                            ├────────────────┼──────────┤                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-32923 │ HIGH     │                   │ 1.7.2, 1.7.2, 1.7.2, 1.5.9 │ vault: Token leases incorrectly treated as non-expiring      │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-32923                   │
│                            ├────────────────┼──────────┤                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-38553 │ MEDIUM   │                   │ 1.8.0                      │ vault: Underlying database file with excessively broad       │
│                            │                │          │                   │                            │ filesystem permissions                                       │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-38553                   │
│                            ├────────────────┤          │                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-38554 │          │                   │ 1.6.6, 1.7.4               │ vault: UI erroneously cached and exposed user-viewed secrets │
│                            │                │          │                   │                            │ between sessions in a...                                     │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-38554                   │
│                            ├────────────────┤          │                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-41802 │          │                   │ 1.7.5, 1.8.4               │ vault: Incorrect Permission Assignment for Critical Resource │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-41802                   │
│                            ├────────────────┤          │                   ├────────────────────────────┼──────────────────────────────────────────────────────────────┤
│                            │ CVE-2021-43998 │          │                   │ 1.7.6, 1.8.5               │ Incorrect Permission Assignment for Critical Resource        │
│                            │                │          │                   │                            │ https://avd.aquasec.com/nvd/cve-2021-43998                   │
└────────────────────────────┴────────────────┴──────────┴───────────────────┴────────────────────────────┴──────────────────────────────────────────────────────────────┘
0s

humblec avatar Nov 15 '22 06:11 humblec

Except for the CVEs in usr/local/bin/cephcsi, you'll need to report them to the Ceph team that builds the base container-image.

The Vault ones look related to the server only, not to the API/client?

nixpanic avatar Nov 15 '22 09:11 nixpanic

Besides mentioned CVEs for Hashicorp, there is a newer one: CVE-2022-36129 (9.1 Critical): https://cve.report/CVE-2022-36129

Fixed in 1.11.1, 1.10.5, and 1.9.8 - https://discuss.hashicorp.com/t/vault-1-11-1-1-10-5-and-1-9-8-released/42389

VladimirMarkelov avatar Nov 22 '22 19:11 VladimirMarkelov

On Tue, Nov 22, 2022 at 11:35:32AM -0800, Vladimir Markelov wrote:

Besides mentioned CVEs for Hashicorp, there is a newer one: CVE-2022-36129 (9.1 Critical): https://cve.report/CVE-2022-36129

Fixed in 1.11.1, 1.10.5, and 1.9.8 - https://discuss.hashicorp.com/t/vault-1-11-1-1-10-5-and-1-9-8-released/42389

This CVE seems only applicable for the Vault server. Ceph-CSI uses the Vault client API only, so it is not affected.

nixpanic avatar Nov 23 '22 08:11 nixpanic

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Dec 23 '22 21:12 github-actions[bot]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

github-actions[bot] avatar Dec 30 '22 21:12 github-actions[bot]

Except for the CVEs in usr/local/bin/cephcsi, you'll need to report them to the Ceph team that builds the base container-image.

It is possible update the packages from the Dockerfile... Updating it in the base image is better and smaller, but that works....

mohag avatar Jan 31 '23 13:01 mohag

Except for the CVEs in usr/local/bin/cephcsi, you'll need to report them to the Ceph team that builds the base container-image.

It is possible update the packages from the Dockerfile... Updating it in the base image is better and smaller, but that works....

@mohag we do that here already ? https://github.com/ceph/ceph-csi/blob/devel/deploy/cephcsi/image/Dockerfile#L31 or you mean something we are missing ?

humblec avatar Jan 31 '23 14:01 humblec

It is possible update the packages from the Dockerfile... Updating it in the base image is better and smaller, but that works....

@mohag we do that here already ? https://github.com/ceph/ceph-csi/blob/devel/deploy/cephcsi/image/Dockerfile#L31 or you mean something we are missing ?

Like that yes, but in the release image (somewhere under line 59) as well (that one is in the build image)

mohag avatar Jan 31 '23 15:01 mohag

@mohag we can do that as well. do you want to submit a PR ? or I can do that. please let me know.

humblec avatar Jan 31 '23 15:01 humblec

@mohag we can do that as well. do you want to submit a PR ? or I can do that. please let me know.

I'll attempt a PR.

mohag avatar Jan 31 '23 21:01 mohag

A big part of the issue with the OS packages here is that the quay.io/centos/centos:8stream image seems to not be routinely updated. (the quay.io/ceph/ceph image uses that as a base) (It should be rebuilt every time an update to a package in that image is available. I could not track down the repo where those Dockerfiles are kept to try and nag them though)

mohag avatar Feb 01 '23 08:02 mohag

This is probably the one you're looking for: https://github.com/tgagor/docker-centos/blob/master/stream8/Dockerfile

Update: Nevermind, this wasn't the original image but an image that has a built-in update.

jeroenlandheer avatar Feb 02 '23 06:02 jeroenlandheer

Let's see if we can get the underlying base images upgraded....

mohag avatar Feb 02 '23 07:02 mohag

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Mar 04 '23 21:03 github-actions[bot]

The base images have been upgraded

mohag avatar Mar 09 '23 07:03 mohag

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Apr 09 '23 21:04 github-actions[bot]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

github-actions[bot] avatar Apr 16 '23 21:04 github-actions[bot]

Can we get this re-opened? Joblib in particular in the ceph-csi image raises a cve score of 9.8 https://avd.aquasec.com/nvd/2022/cve-2022-21797/

Seems potentially worth looking at since it's vulnerable to arbitrary code execution. I'm running v3.11.0 which seems to be the newest version, and is still vulnerable to this.

Starttoaster avatar Apr 10 '24 17:04 Starttoaster

Can we get this re-opened? Joblib in particular in the ceph-csi image raises a cve score of 9.8 https://avd.aquasec.com/nvd/2022/cve-2022-21797/

Seems potentially worth looking at since it's vulnerable to arbitrary code execution. I'm running v3.11.0 which seems to be the newest version, and is still vulnerable to this.

Considering most of the vulnerabilities are in base image, thats the place we have to look into

humblec avatar Apr 11 '24 03:04 humblec

If it's not a necessary dependency, the option of uninstalling it from the image in the Dockerfile here is an option that is available as well. If it is a necessary dependency, there's not much to really do (maybe look for alternatives) since there's apparently no patched version.

I understand that the maintainers of ceph-csi might find that to be less than a "clean" solution. But ceph-csi is a product that is only expected to be ran in a container. So hardening the production image everyone uses seems like it shouldn't be an incredibly tall order for this project to take on, imho, even if it's a stopgap to getting the fix in the upstream base image. Certainly not such a tall order that a 9.8 score CVE stays in the production image for a year and a half (the approximate age of this issue.)

I'd even be happy to try my hand at helping contribute this fix if the maintainers here are open to the fix being implemented here. I mean... it's an arbitrary code execution vulnerability in an image running as root in my clusters with host mode networking and touches my storage clusters. I feel like that sounds like a pretty important thing to tighten up. If I'm being melodramatic let me know, but it seems like something worth acting on last year. That all being said, I'm extremely grateful for the tool, both this cluster client, and ceph in general are amazing. Absolutely wanted to underscore I'm not undermining the awesomeness of it, I just want it to be awesome and actually reasonably secure to run.

Starttoaster avatar Apr 11 '24 03:04 Starttoaster

If ceph-csi's maintainers are dead set on the vulnerability resolution being implemented in the upstream container, can we get a link to the Issue tracking it upstream over here? I'd very much appreciate it!

Starttoaster avatar Apr 11 '24 19:04 Starttoaster

@Starttoaster Issues like : https://github.com/ceph/ceph-container/issues/2077 try to cover this request.

humblec avatar Apr 13 '24 13:04 humblec

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar May 13 '24 21:05 github-actions[bot]

Still not stale. Relevant here but still tracking in ceph-container as well

Starttoaster avatar May 16 '24 19:05 Starttoaster

This might be somewhat solved now. There's still a critical vulnerability but now it's just from slightly out of date Go dependencies, since this switched to a base CentOS Stream 9 image. Thanks @Madhu-1 !!

Starttoaster avatar Aug 17 '24 00:08 Starttoaster

Closing this one as we have updated to use centos 9 image. @Starttoaster Thanks for checking, feel free to open issue for go dependencies :)

Madhu-1 avatar Aug 19 '24 08:08 Madhu-1