opencti icon indicating copy to clipboard operation
opencti copied to clipboard

Handle pending files better to prevent UI latency issues

Open rlynch-ironnet opened this issue 2 years ago • 0 comments

Description

In response to the discussion at https://luatix.slack.com/archives/CHZC2D38C/p1663345872905849 pending files needs a fix of some sort.

TLDR: An error in configuration caused ~3000 pending files from the mandiant connector to be created. Because the current Minio path is effectively import/pending/<file_name> with the entity_id being a key in the metadata, ALL 3000 files end up getting a head-object API call before the UI finishes rendering the view while it waits on the graphql pendingFiles call.

Understanding that a single entity can have multiple files, uploading as import/pending/<entity_id> isn't feasible, but what about import/pending/<entity_id>/<file_name>? Or could this potentially result in orphaned files if the entity_id changes?

Alternatively, the pendingFiles GQL could be split off from the main GQL call, and pending files gets populated whenever the call finishes.

I'm not sure theres a good solution here, I'm mostly creating this for tracking that an issue exists.

Environment

  1. aws fargate
  2. 5.3.12
  3. frontend
  4. n/a

Reproducible Steps

Make a whole bunch of pending files, watch it explode.

Expected Output

UI is rendered in a timely manner

Actual Output

UI is rendered partially, then the observable data is rendered ~20s later

Additional information

n/a

Screenshots (optional)

n/a

rlynch-ironnet avatar Sep 19 '22 14:09 rlynch-ironnet