Onpremise kafka and zookeeper have outdated java jre/jdk

[Marc-Leidos]

Problem Statement

The self-host/onpremise Sentry deploys two containers:

image: "confluentinc/cp-zookeeper:5.5.0" image: "confluentinc/cp-kafka:5.5.0"

that both use an outdated Java 8 version (openjdk version "1.8.0_222") Because copies of the java executable appear in /var/lib/docker/overlay2 they are picked up by Nessus scan and flagged because the are less than Java 1.8.0_302.

I've tried directly updating the cp-zookeeper and cp-kafka image versions, but anything over 5.5.7 doesn't update the java version to the level needed. Trying the latest version 7.0.1 breaks Sentry and it won't start it appears that uses Java 11.

Solution Brainstorm

Moving to an updated kafka/zookeeper image using the l latest Java 8 would help solve the immediate problem. A solution for clearing the old file versions kept in /var/lib/docker/overlay2 would still have to be worked out.

[chadwhitacre]

Seems like a reasonable request on its face, @Marc-Leidos, thanks for surfacing. Putting on the backlog for now to collect interest/discussion. One initial thought is that this might be parallel to #1097, where we would want to understand where Sentry SaaS is at and what the timeline for upgrading is over there. Upgrading in self-hosted would be experimental until SaaS upgrades.

[christophecalus]

Our Nessus scanner flagged this issue too, but also the log4j 1.2 that is used in the kafka image. It has multiple vulnerabilities and is outdated/unsupported. Not the big one that is introduced when using a 2.x - <2.16 version, but still important. List of reported vulnerabilities: CVE-2019-17571, CVE-2020-9488, CVE-2022-23302.

[aminvakil]

@christophecalus Thanks for bringing these up.

About CVE-2019-17571, please take a look at https://github.com/getsentry/self-hosted/issues/1219.

But about the others AFAIK Sentry SaaS should be upgraded first (if sentry kafka / zookeeper with its specific configurations is affected).

Also we should make sure first that our current kafka / zookeeper (confluentinc/cp-kafka:5.5.0 / confluentinc/cp-zookeeper:5.5.0) is affected and if it will be fixed by upgrading to a recent release of https://github.com/confluentinc/kafka.

Or we should make some configuration change to mitigate the vulnerabilities.

[christophecalus]

@christophecalus Thanks for bringing these up.

About CVE-2019-17571, please take a look at #1219.

But about the others AFAIK Sentry SaaS should be upgraded first (if sentry kafka / zookeeper with its specific configurations is affected).

Also we should make sure first that our current kafka / zookeeper (confluentinc/cp-kafka:5.5.0 / confluentinc/cp-zookeeper:5.5.0) is affected and if it will be fixed by upgrading to a recent release of https://github.com/confluentinc/kafka.

Or we should make some configuration change to mitigate the vulnerabilities.

Hi @aminvakil,

I was checking the issue https://github.com/getsentry/self-hosted/issues/1219 and followed the link to the kafka repo. It is fixed by replacing the log4j-1.2.17.jar with a patched log4j-1.2.17-cp1.jar file. But this isn't done in version 5.5.0, you can even see in version 5.5.7 it has been replaced with a log4j-1.2.17-cp2.jar file. I think it would be best to at least use the most recent 5.5.x version of the kafka docker image, but of course as you mentioned, the other CVE's & openJDK version needs to be checked too.

[aminvakil]

Hi @aminvakil,

I was checking the issue #1219 and followed the link to the kafka repo. It is fixed by replacing the log4j-1.2.17.jar with a patched log4j-1.2.17-cp1.jar file. But this isn't done in version 5.5.0, you can even see in version 5.5.7 it has been replaced with a log4j-1.2.17-cp2.jar file.

Correct. @chadwhitacre Could we reopen the #1219 ?

I think it would be best to at least use the most recent 5.5.x version of the kafka docker image,

I think so too.

but of course as you mentioned, the other CVE's & openJDK version needs to be checked too. Yeah, I also found CVE-2022-23302 in https://logging.apache.org/log4j/1.2/ as won't be fixed (although CVE-2019-17571 is been listed there too, but it has been in kafka).

~cc @getsentry/security~ Looks like this doesn't work.

@chadwhitacre Could you please mention this to security guys about what should we do and how can we help?

[aminvakil]

Thanks to @GTB3NW report in https://github.com/getsentry/self-hosted/issues/1445, seems like there is another problem using outdated java, I have tested 5.5.9 (using jdk 1.8.0_222) and the issue has not been fixed either, 6.0.0 does not have the problem but the zookeeper containers fails to start with this error:

[fedora@fedora self-hosted]$ docker-compose ps
NAME                              COMMAND                  SERVICE             STATUS              PORTS
sentry-self-hosted-clickhouse-1   "/entrypoint.sh"         clickhouse          exited (0)          
sentry-self-hosted-kafka-1        "/etc/confluent/dock…"   kafka               created             
sentry-self-hosted-redis-1        "docker-entrypoint.s…"   redis               exited (0)          
sentry-self-hosted-zookeeper-1    "/etc/confluent/dock…"   zookeeper           exited (143)        
[fedora@fedora self-hosted]$ docker-compose logs zookeeper
sentry-self-hosted-zookeeper-1  | ===> User
sentry-self-hosted-zookeeper-1  | uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
sentry-self-hosted-zookeeper-1  | ===> Configuring ...
sentry-self-hosted-zookeeper-1  | ===> Running preflight checks ... 
sentry-self-hosted-zookeeper-1  | ===> Check if /var/lib/zookeeper/data is writable ...
sentry-self-hosted-zookeeper-1  | ===> Check if /var/lib/zookeeper/log is writable ...
sentry-self-hosted-zookeeper-1  | ===> Launching ... 
sentry-self-hosted-zookeeper-1  | ===> Launching zookeeper ... 
sentry-self-hosted-zookeeper-1  | [2022-06-18 11:02:37,273] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
sentry-self-hosted-zookeeper-1  | [2022-06-18 11:02:37,576] WARN o.e.j.s.ServletContextHandler@1750fbeb{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
sentry-self-hosted-zookeeper-1  | [2022-06-18 11:02:37,577] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)

@chadwhitacre Could you please mention this internally again?

[bulldoza999]

definitely would love to help on this if possible.. Is there a way to update via an env variable like

i see the issue and on further review i believe we have to add the additional env variables

ZULU_OPENJDK_VERSION8: 8.62.0.19 - which i confirmed is the latest from the zulu openjdk java version

pulled that reference from : https://hub.docker.com/layers/cp-kafka/confluentinc/cp-kafka/5.5.10/images/sha256-64af7e36aee8dd921685d9033ea44ca5f7d2639a047f6129706f4c7ae3947300?context=explore

I can also verify 5.5.10 does work on the latest 22.6.0 on my self service release if that helps (with no changes apart from updating the docker-compose.yml)

[instinct-vfx]

Was there any tests already to jump directly to the current versions? As the other issue was closed in favor of this one. We need to drop log4j 1.x, the most recent versions seem to do that, but we are unsure if that is possible at all.

[aminvakil]

Another user (https://github.com/getsentry/self-hosted/issues/1704) hit this, so I want to nag about this again :sweat: