opencti icon indicating copy to clipboard operation
opencti copied to clipboard

[backend] Improve Attack Pattern knowledge matrix view performance (#6662)

Open SouadHadjiat opened this issue 1 year ago • 1 comments

Proposed changes

  • optimize attackPatterns query (use aggregations for isSubAttackPattern), use batch loading for parentAttackPatterns and subAttackPatterns
  • add pagination and filters for sub connections (coursesOfAction, parentAttackPatterns, subAttackPatterns), because we were only retrieving the first 20, without any way to retrieve more.
  • create a dedicated and optimized query for attack pattern matrix view : attackPatternsMatrix
  • TODO : code cleanup, renaming, frontend refactoring ?

Related issues

  • #6662

Checklist

  • [x] I consider the submitted work as finished
  • [x] I tested the code for its functionality
  • [ ] I wrote test cases for the relevant uses case (coverage and e2e)
  • [ ] I added/update the relevant documentation (either on github or on notion)
  • [ ] Where necessary I refactored code to improve the overall quality

Further comments

After first 2 commits, perf test with +2k attack patterns (exported from testing)

  • on master: query takes 20s Capture d'écran 2024-08-26 160150

  • on branch: query takes 7s image

  • on branch - after replacing with attackPatternsMatrix query : query takes less than 1s Capture d'écran 2024-08-28 114752

SouadHadjiat avatar Aug 01 '24 17:08 SouadHadjiat

Codecov Report

Attention: Patch coverage is 45.26316% with 104 lines in your changes missing coverage. Please review.

Project coverage is 66.31%. Comparing base (579e409) to head (9de5539). Report is 7 commits behind head on master.

Files with missing lines Patch % Lines
.../opencti-graphql/src/database/middleware-loader.ts 18.18% 45 Missing :warning:
...ti-platform/opencti-graphql/src/database/engine.js 4.87% 39 Missing :warning:
...atform/opencti-graphql/src/domain/attackPattern.ts 72.22% 20 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7924      +/-   ##
==========================================
+ Coverage   66.14%   66.31%   +0.16%     
==========================================
  Files         597      597              
  Lines       59936    60396     +460     
  Branches     6154     6253      +99     
==========================================
+ Hits        39646    40051     +405     
- Misses      20290    20345      +55     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Aug 26 '24 09:08 codecov[bot]