Referential integrity check does not complete with large bundles.

[jstone-psi]

🐛 Describe the bug

When validating a large bundle, there is a risk that the referential integrity check will not complete, due to an out of memory error. This happened recently with the spacewatch bundle, which has 1.8 million products.

📜 To Reproduce

Steps to reproduce the behavior:

Run validate on the spacewatch bundle: validate -D -R pds4.bundle -t gbo.ast.survey.spacewatch

spacewatch_labels.txt.gz

I've attached the output from our last validation run which illustrates the error, as well.

🕵️ Expected behavior

The referential integrity check should have completed, and raised errors in the bundle related to a LID collection id mismatch. At minimum, a more clearly marked abnormal exit would be desirable, so that it's clear that the validation results cannot be used.

📚 Version of Software Used

Validate 2.1.4

🩺 Test Data / Additional context

If needed, I can provide a stub bundle with all of the spacewatch labels.

🏞Screenshots

🖥 System Info

• OS: CentOS7

---

🦄 Related requirements

⚙️ Engineering Details