f3d icon indicating copy to clipboard operation
f3d copied to clipboard

Malformed .mdl files can cause a large memory leak

Open iommu opened this issue 3 weeks ago • 9 comments

Describe the bug Malformed .mdl files can cause an extremely fast memory leak. Grows to >10gb in ~2s on my computer before my computer freezes and the OOM killer kicks in.

To Reproduce Steps to reproduce the behavior:

Type 1 (extreme memory leak):

  1. Create a malformed file which triggers the leak
version https://git-lfs.github.com/spec/v1
oid sha256:eeefab2ffdfdf898c666f1ed79a2ff0b74557558230e14d2f051db9f158dce3d
size 124748

named as an .mdl file e.g. leak.mdl 2. Either open the file in the F3D app or access the file with your preferred GUI file browser triggering the thumbnailer WARNING : this will cause your computer to run out of ram and either crash or trigger the OOM if you have one 3. Large memory leak

Type 2 (immediate F3D crash)

  1. Create a malformed file which triggers a crash

(intentionally left blank) named as an .mdl file e.g. crash.mdl 2. Open the file in the f3d app 3. Immediate F3D crash

Expected behavior Program should not memory leak and not crash and instead safely error out and report error (this is especially so for the thumbnailer).

System Information:

Running on Arch Linux (BTW)

F3D Information

f3d --version
F3D 3.3.0

F3D - A fast and minimalist 3D viewer
Version: 3.3.0.
Build date: 2025-10-25 17:11:50.
Build system: Linux 64-bits.
Compiler: GNU 15.2.1.
Module ImGui: ON.
Module OpenEXR: ON.
Module Raytracing: ON.
Module WebP: OFF.
VTK version: 9.5.2.
Copyright (C) 2019-2021 Kitware SAS.
Copyright (C) 2021-2025 Michael Migliore, Mathieu Westphal.
License BSD-3-Clause.

Additional context

iommu avatar Dec 09 '25 06:12 iommu

I found this issue through opening Gnome Nautilus to the f3d/testing/data directory after cloning the project without git LFS (hence the provided file content) where Nautilus would crash because the F3D thumbnailer would trigger the above memory leak and trigger the OOM.

iommu avatar Dec 09 '25 07:12 iommu

The MDL importer definitely is not safe and we definitely need to fix that.

The format itself makes it hard but I'm sure we can do better. Thanks for opening the issue. I reproduce both issues.

mwestphal avatar Dec 09 '25 07:12 mwestphal

I searched through all the f3d*.thumnailers in /usr/share/thumbnailers and found

  1. only the f3d-plugin-native.thumbnailer caused a memory leak
  2. only the MimeType=application/vnd.mdl in f3d-plugin-native.thumbnailer caused a memory leak

iommu avatar Dec 09 '25 07:12 iommu

only the MimeType=application/vnd.vtm in f3d-plugin-native.thumbnailer caused a memory leak

I'm not following, your issue mention .mdl files, not .vtm files, did you found another issue ?

mwestphal avatar Dec 09 '25 07:12 mwestphal

ah sorry I coppied the wrong line, yes MimeType=application/vnd.mdl crashed sorry

iommu avatar Dec 09 '25 07:12 iommu

ah sorry I coppied the wrong line, yes MimeType=application/vnd.mdl crashed sorry

alright, it makes sense.

Well, feel free to take a look into it if you want :)

mwestphal avatar Dec 09 '25 07:12 mwestphal

The MDL importer definitely is not safe and we definitely need to fix that.

The format itself makes it hard but I'm sure we can do better. Thanks for opening the issue. I reproduce both issues.

Yeah I definitely don't envy the position of trying to fix this, it sounds like a hard problem. However maybe MimeType=application/vnd.mdl files should be removed from the f3d-plugin-native.thumbnailer until this is fixed is a good idea, as the possibility of freezing your computers upon accessing a directory and triggering the thumbnailer is a bit of a problem.

iommu avatar Dec 09 '25 07:12 iommu

Well, feel free to take a look into it if you want :)

Sure! All thought I don't have much experience with file parsers so I don't know how far I'll get :<

iommu avatar Dec 09 '25 07:12 iommu

as the possibility of freezing your computers upon accessing a directory and triggering the thumbnailer is a bit of a problem.

Agreed, lets try to fix the mem leak before the next release

mwestphal avatar Dec 09 '25 07:12 mwestphal