opencti
opencti copied to clipboard
Handle pending files better to prevent UI latency issues
Description
In response to the discussion at https://luatix.slack.com/archives/CHZC2D38C/p1663345872905849 pending files needs a fix of some sort.
TLDR: An error in configuration caused ~3000 pending files from the mandiant connector to be created. Because the current Minio path is effectively import/pending/<file_name>
with the entity_id being a key in the metadata, ALL 3000 files end up getting a head-object API call before the UI finishes rendering the view while it waits on the graphql pendingFiles call.
Understanding that a single entity can have multiple files, uploading as import/pending/<entity_id>
isn't feasible, but what about import/pending/<entity_id>/<file_name>
? Or could this potentially result in orphaned files if the entity_id changes?
Alternatively, the pendingFiles GQL could be split off from the main GQL call, and pending files gets populated whenever the call finishes.
I'm not sure theres a good solution here, I'm mostly creating this for tracking that an issue exists.
Environment
- aws fargate
- 5.3.12
- frontend
- n/a
Reproducible Steps
Make a whole bunch of pending files, watch it explode.
Expected Output
UI is rendered in a timely manner
Actual Output
UI is rendered partially, then the observable data is rendered ~20s later
Additional information
n/a
Screenshots (optional)
n/a