BUG | memleak

[spirillen]

What is the problem you are experiencing?

Was trying to report this as Report a security vulnerability

but all I got is this

For the first time in years, I see what appears to be a memory leak, but as the system was completely unresponsive, and can't do much more than sharing a screenshot of the memory usage for until I had to increase the amount of RAM assigned to the mashine

How can we reproduce the problem?

To reproduce the problem:

1. install the ProxMox virtual machine https://github.com/if-you-want-peace-prepare-for-war/proxmox/blob/master/ct/pyfunceble.sh (Not fully up to date with running version)
2. run the command pyfunceble -c --preload --all --execution --hierarchical -f https://raw.githubusercontent.com/mypdns/matrix/refs/heads/master/source/porn_filters/explicit_content/wildcard.list
3. Wait over the night, and this is the result.
4. PS: the --preload are extremely slow to write in to the DB, only about 12 records per seconds, should be more like 1000-3000 depending on methods (PostgreSQL)

Do you have a screenshot?

Screenshot

What did you expect to happen?

Not using more than 100 mb of RAM

Is there a workaround?

Not that I've found

How did you configure PyFunceble?

cli_testing:
  inactive_db: yes
  whois_db: yes
  db_type: postgresql
lookup:
  dns: yes
  http_status_code: yes
  netinfo: yes
  special: yes
  whois: yes
  reputation: no
  platform: yes
  timeout: 2
dns:
  follow_server_order: no
  trust_server: yes
  server:
    - 9.9.9.10
    - 149.112.112.10
  protocol: UDP
  delay: 0.0

verify_ssl_certificate: no
max_http_retries: 2

platform:
  push: yes
  preferred_status_origin: recommended
  checker_priority:
    - availability
    - syntax
  checker_exclude:
    - reputation

Which Operating System did you use?

Ubuntu 24.04 LTS

Which Python Version did you use?

3.11

Which PyFunceble Version did you use?

4.3.0a4.dev

Additional context

I would like to add...

[spirillen]

Reproduced...

[spirillen]

It is easy to see when PyF are running both standard tests + platform

[spirillen]

This is interesting... it seems like the memleak is related to the PostdreSQL. In the following screenshot I have solely been using MariaDB

[funilrys]

The URL is correct: https://github.com/funilrys/PyFunceble/security/advisories

[spirillen]

Have seen traces of this memleak, when using both MariaDB and csv. This is in way smaller scale, yet the memory usage are climbing steadily

[funilrys]

Can you please retest this with a newer 4.3.0aXX version ? I tried to reproduce it and couldn't.

[spirillen]

New test started

[spirillen]

[spirillen]

[spirillen]

Process hung

Mar 26 01:56:08 pyfunceble-01 python3[42556]: 18hdxxx.com                                                                                          VALID       SYNTAX     26. Mar 2025 00:54:41
Mar 26 01:56:08 pyfunceble-01 python3[42556]: 0redird.com                                                                                          VALID       SYNTAX     26. Mar 2025 00:54:41



root@pyfunceble-01:~# systemctl restart pyfunceble.service ; journalctl -fu pyfunceble.service
Mar 26 01:56:07 pyfunceble-01 python3[42556]: xy.nullrefexcep.com                                                                                  VALID       SYNTAX     26. Mar 2025 00:54:41
Mar 26 01:56:07 pyfunceble-01 python3[42556]: 0x01n2ptpuz3.com                                                                                     VALID       SYNTAX     26. Mar 2025 00:54:41

[spirillen]

And then it created this one... 💤 Have we encountered that once or twice before? Nevertheless, you can now see that the memory usage is steadily increasing over time, raising with around ~100 to ~150 MB per hour.

For good measure, the database is hosted on separate hardware apart from the virtual machine.

[spirillen]

Restart Pyf platform. Once again.. it is using more and more RAM over time until it hit 2.5 - 3 GB, but properly continues as first reported.

[funilrys]

Closing since we made so many changes since March.

Added an item for this in the Backlog.