Checkov run with Prisma API arguments causes OOM

[zagr0]

Describe the issue We use Prisma cloud and run checkov jobs with Gitlab k8s runners on VM with 4 CPU cores and 16Gb of RAM for our infrastructure repository (mono repo), it contains terraform plans, ansible automations, helm charts, kustomize configurations. When we use Prisma API url and access keys arguments to report result to Prisma, checkov job always crushes, it's terminated by OOMkiller as chechov process consumes all the available memory on the node. The interesting thing is that if we run checkov without prisma integration it runs well and not OOM killed, no such memory consumption. Without Prisma arguments it takes ~4-5Gb of RAM, which is also pretty a lot actually.

Examples Runs with Prisma, OOM: checkov -d . --repo-id our/repo-id --branch branch_name --prisma-api-url https://api.prismacloud.io --bc-api-key XXXXXXXXXXXXXX::YYYYYYYYYYYYYY --use-enforcement-rules -o junitxml But runs well without Prisma: checkov -d . -o junitxml

Exception Trace There is no checkov errors, it just ate all the memory and killed:

ERROR: Job failed (system failure): Error in container build: exit code: 137, reason: 'OOMKilled'

Desktop (please complete the following information):

• Runs in GKE 1.30
• Checkov Version 3.2.356

Additional context Not sure but probably started to happen on v3, before we didn't face the issue.

[tsmithv11]

Hey @zagr0 since you are a Prisma Cloud customer, can you work with support to get a case opened for this? Then engineering can take a look

[robinsmidsrod]

Any movement on this issue?

After some trial and error I found out that running the following frameworks on our big FluxCD monorepo is causing OOM, so I've just had to skip them or we'll always get a crash:

  --skip-framework ansible \
  --skip-framework argo_workflows \
  --skip-framework bitbucket_pipelines \
  --skip-framework github_actions \
  --skip-framework gitlab_ci \
  --skip-framework json \
  --skip-framework kubernetes \
  --skip-framework kustomize \
  --skip-framework yaml \

It should at least help you to narrow things down to which scanners that are causing problems.

[robinsmidsrod]

And when I look at that list from a bit of a distance, it dawns on me that they are all based on scanning JSON and/or YAML files. Is it possible that you read in all the content of all those structured files into one big (memory) buffer, instead of allocating memory for one file at a time? And if this is also duplicated per framework, then it could cause this kind of issue.

[zagr0]

Hi @robinsmidsrod , from my side I have opened additional case to the Prisma Cloud Support, they are still investigating. Seems the issue for us also related to our ArgoCD and kustomize configuration directory, appeared after v2 -> v3 upgrade, probably when json/yaml parser was changed.

[maxamel]

@zagr0 is this still relevant or can be closed ?

[zagr0]

Yes, the issue is till there

[stale[bot]]

Thanks for contributing to Checkov! We've automatically marked this issue as stale to keep our issues list tidy, because it has not had any activity for 6 months. It will be closed in 14 days if no further activity occurs. Commenting on this issue will remove the stale tag. If you want to talk through the issue or help us understand the priority and context, feel free to add a comment or join us in the Checkov slack channel at codifiedsecurity.slack.com Thanks!

[robinsmidsrod]

The issue is still present.