Add useful Filters into the import-external-reference connector

[lhviet]

Use case

The current configurations of the connector import-external-reference doesn't allow us to disable the PDF and MD generation for selected observables. While 99% of generated PDFs and MDs are blank, the generated stuffs are useless and they cost us.

• For example, everyday, we ingest 700,000 new observables
• Then the connector tries to generates 700K (most of them are blank) PDF files and store into our storage & database

Those blank files are useless, but they cost us expensive in

• storage, i.e., AWS S3
• backup/snapshot service and storage
• traffic/requests (GET/POST/PUT, etc.)
• CPU/RAM, or slow down the system because of unnecessary data

Current Workaround

Proposed Solution

I would suggest a few possible solutions below:

• An option to prevent blank files generated
• An option to not process specified Labels (label-based filters)

Additional Information

Would you be willing to submit a PR?

We strongly encourage you to submit a PR if you want and whenever you want. If your issue concern a "Community-support" connector, your PR will probably be accepted after some review. If the connector is "Partner-support" or "Filigran-support", a dev team make take over but will base its work on your PR, speeding the process. It will be much appreciated.