feat(core): Save memory at execution

[cr0hn]

Context

This PR has a new feature that significantly reduces the memory footprint when Prowler is running.

Description

This PR aims to reduce Prowler's memory footprint by switching from using the regular Python list and dictionaries to using SQLite3 as a database.

In the first place, it could sound a bit weird, but it's actually a good idea to reduce Prowler's memory footprint, and the performance impact is really low.

In some tests I've run, Prowler's memory footprint went from 5GB to 400-500MB, which is a huge improvement.

In this PR not all checks of all providers are implemented, but only these ones:

• AWS

And for the following AWS services:

• CloudWatch
• S3
• EC2
• ECS
• CloudWatch Logs
• SageMaker
• Lambda
• SSM

In addition, this PR also includes a quick fix that aims to use Prowler with local AWS emulators (e.g., localstack or moto).

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jfagoagas]

Hello @cr0hn, this is great! We will review as soon as we can, in the meantime could you merge your branch with the master branch since it seems that some things changed since the point your branch was created.

Thanks!

[jfagoagas]

@cr0hn could you please review the errors raised from our formatters here https://github.com/prowler-cloud/prowler/actions/runs/10773270870/job/29872693660?pr=4892?

[cr0hn]

I just fixed the linter errors @jfagoagas

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 82.97101% with 94 lines in your changes missing coverage. Please review.

Project coverage is 88.83%. Comparing base (5e3da2d) to head (41d7210).

Files with missing lines	Patch %	Lines
prowler/__main__.py	0.00%	43 Missing :warning:
prowler/lib/persistence/sqlite.py	92.63%	14 Missing :warning:
prowler/lib/outputs/outputs.py	33.33%	12 Missing :warning:
prowler/lib/persistence/manager.py	60.00%	12 Missing :warning:
prowler/lib/utils/gc.py	0.00%	12 Missing :warning:
prowler/lib/check/check.py	66.66%	1 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4892      +/-   ##
==========================================
- Coverage   88.90%   88.83%   -0.08%     
==========================================
  Files         946      950       +4     
  Lines       29053    29441     +388     
==========================================
+ Hits        25831    26153     +322     
- Misses       3222     3288      +66     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[jfagoagas]

I'm closing this PR as we've ultimately decided to take a different direction. Huge thanks for the effort and work put into this, @cr0hn — it's truly appreciated!