nexB
Data Import from NVD data via GitHub - Script Added
Summary
This pull request enhances the vulnerability data import by allowing data to be imported from a json data stored in the local directory NVD_Data. It also considers robust error handling to manage duplicate package entries and missing data. Additionally, the code supports storing a simplified "plain" package URL without qualifiers or subpaths.
Related Issues
https://github.com/aboutcode-org/vulnerablecode/issues/1437
Testing Instructions
Run the import process with the following command:
python manage.py import_data
Test importing data from a local folder i.e NVD_Data.
Check that duplicate package entries are gracefully handled and skipped. Review the logs to ensure they accurately reflect each step of the process, including any errors or skipped files. Verify that plain package URLs (without qualifiers and subpaths) are being correctly stored in the database.
Changes With New commit
Now NVD data is not included in the code directory I can directly import the Data by using Git Tree REST API and will import data from whole 6400 files.
I have tried to import the data directly from the github api but that was quite a slow process and also it was not importing full data from almost 6400 files it was able to import from 1800 files.Maybe there is rate limit problem of github api. So what would you recommend to do in this situation.
@pombredanne The recent changes are working fine you just have to put your GitHub PAT in the code and you can run python manage.py import_nvd_data the import will start and when I have runned the script to import the data on my local machine then almost cumulatively 25k data entries were created in the Package Related Vulnerabilities , package, vulnerability,vulnerability reference Models.