fix: Command Getting Killed While Updating CVE Database or Scanning with cve-bin-tool

[pooja0805]

Description

Hello Team,

I encountered an issue when attempting to update the CVE database or scan a file/directory using the cve-bin-tool. The process terminates with a "Killed" error, as shown in the output below:

[test@f961efb782bb ~]$ cve-bin-tool --disable-data-source EPSS -u now  
[03:45:39] INFO     cve_bin_tool - CVE Binary Tool v3.4                      cli.py:624  
           INFO     cve_bin_tool - This product uses the NVD API but is not  cli.py:625  
                    endorsed or certified by the NVD.  
           INFO     cve_bin_tool - For potentially faster NVD downloads,     cli.py:628  
                    mirrors are available using -n json-mirror  
           INFO     cve_bin_tool - Disabling data source EPSS                cli.py:771  
[03:45:49] WARNING  cve_bin_tool.CVEDB - Updating cachedir                 cvedb.py:652  
                    /home/test/.cache/cve-bin-tool  
[03:45:52] INFO     cve_bin_tool - Getting NVD CVE data...            nvd_source.py:389  
           INFO     cve_bin_tool - Getting GitLab Advisory Database    gad_source.py:86  
                    CVEs...  
           INFO     cve_bin_tool - Getting RedHat CVEs...           redhat_source.py:69  
           INFO     cve_bin_tool - Getting PURL2CPE data...       purl2cpe_source.py:36  
Downloading CVEs... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:07  
[03:49:15] INFO     cve_bin_tool - Adding 35170 RedHat CVE entries redhat_source.py:136  
[03:51:30] INFO     cve_bin_tool - Getting Open Source Vulnerability  osv_source.py:161  
                    Database CVEs...  
Killed  

The command terminates unexpectedly after downloading and processing some of the CVE data.

To reproduce

Steps to reproduce the behaviour: Run the following command to update the CVE database:

cve-bin-tool --disable-data-source EPSS -u now 

Expected behaviour: The command should complete the update process or scan successfully without terminating unexpectedly. Actual behaviour: The command terminates unexpectedly after downloading and processing some of the CVE data.

Version/platform info

Version of CVE-bin-tool( e.g. output of cve-bin-tool --version): 3.4 Installed from pypi or github? pypi Operating system: Linux/x86-64, linux/ppc64le Python version (e.g. python3 --version): Python 3.11 Running in any particular CI environment we should know about? NA

Anything else?

Is there a known solution for this issue? Could this be related to resource limitations or another underlying problem? Any guidance or suggestions would be appreciated.

Thank you!

[terriko]

"killed" usually means it's been done by your system (cve-bin-tool itself terminates differently), so you've probably exceeded some system limit.

Likely culprits:

• timeout (this happens a lot in CI and cloud systems with limits imposed)
• disk space full

Running -u now will take about 10-30 minutes (depending on network/disk speed) and will need more than 2.5G of disk space.

BTW, even once you get things sorted I think cve-bin-tool will fail in your command because there's no file to scan, but it should do so after updating the database.

[pooja0805]

@terriko Thanks for your quick response. Let me check my system configurations and try out again.

[shivamv300]

hi @terriko I'm also running into the same issue, even though I have 50+GB available disk space. I tried running the scan in batches (by disabling some data sources: --disable-data-source NVD,OSV,PURL2CPE,REDHAT,RSD in the first pass), but the first pass itself has been running for more than 10 hours. (My system has 4 cores, and 16 GB memory - out of which 14+ GB is free).

I did have an observation with this first pass scan:htop shows resource utilisation for one of the CPU cores close to 99% (while the other cores are free)

[terriko]

We've been seeing some out of memory issues that are likely related: #4710

@shivamv300 Other people have reported that disabling OSV solves the problem, but I see you've already tried that. The other thing that works in some cases is upgrading to python 3.13 (which has other memory improvements). I'm hideously sick so I haven't been able to work on figuring out a root cause, so any info and analysis is very welcome!

[visuman]

hi @terriko I'm also running into the same issue, even though I have 50+GB available disk space. I tried running the scan in batches (by disabling some data sources: --disable-data-source NVD,OSV,PURL2CPE,REDHAT,RSD in the first pass), but the first pass itself has been running for more than 10 hours. (My system has 4 cores, and 16 GB memory - out of which 14+ GB is free).

I did have an observation with this first pass scan:htop shows resource utilisation for one of the CPU cores close to 99% (while the other cores are free)

@terriko I am also facing the same issue as @shivamv300 . Have we figured out a fix or alternative for this other than disabling OSV and upgrading to python 3.13?

[terriko]

We don't have a solid fix yet -- the workarounds are still "get more RAM, disable OSV, upgrade to python 3.13"

There are some people working on a bunch of harder bugs this week, though, and this is one on their todo list.

[terriko]

Also, I don't have minimum ram numbers, but at two points of data, apparently it will consistently get killed at 8gigs of ram, but I've never been able to duplicate the bug on my system with 32g. So the minimum amount is somewhere between the two at the moment.

[Luchostein]

Hi @terriko. You may try ulimit -m 1000000 or so to force the memory limitation and reproduce the bug.

[arkcoreio]

Had the same problem on different EC2's the i changed the command to below. cve-bin-tool -d EPSS, OSV --nvd-api-key This command worked on a tiny EC2 with 40GB disk without any problems.