Crashes with deleting memory allocated in dlls

[drussel]

Overall, we have been quite happy with mimalloc except on windows with applications that use qt. There we get random crashes. The most consistent pattern is in debug builds when hitting the destructor of a std::string that was created within a Qt dll (eg the QString::toStdString). Such crashes can have various cases such as mismatched runtimes, but I think we don't have that. And disabling mimalloc makes it go away.

We both link in the override dll and include mimalloc-new-delete.h as described in https://microsoft.github.io/mimalloc/overrides.html.

It is not clear to me how that would work in general (how it would intercept new/delete calls in the Qt dll), but my knowledge of the inner workings of dlls is very minimal.

Version 1.6.3 and 1.6.1. Building with make. Visual studio 2017 (various revisions including the most recent).

[daanx]

Thank you for the report -- I hope we can fix this.

I just searched for Qt and custom allocators and found some benchmarks [1] by @vohi where he compared against mimalloc (and other allocators) so it seems in principle mimalloc should be fine -- but this was done in Linux. I guess the issue on Windows only.

• Is there a way you can provide detailed instructions so I can build and try to reproduce the problem locally?
• Otherwise, is it possible that you can build mimalloc with cmake -DMI_DEBUG_FULL=ON and running it with the environment variable set MI_MALLOC_VERBOSE=3 ? -- this will show at the start all mimalloc options and, for Windows, redirection messages. Hopefully, it will show a warning when it goes wrong; It would be great if you could at that point choose the VS debugger and show a stack trace. (and try to use the latest mimalloc, 1.6.3, that will also detect heap block overflows in case of a Qt bug)
• Otherwise, perhaps it is related to not capturing the right new/delete overloads??; you could try to build without using mimalloc-new-delete and try that?

Hope we can find a way to reproduce and fix this, Best, Daan

[1] see https://www.qt.io/blog/a-fast-and-thread-safe-pool-allocator-for-qt-part-1

[drussel]

Nice. Getting rid of the mimalloc-new-delete.h include makes the problem go away. I should have tried that :-)

Is there a way to get more information about missing overloads? Given that, is it worth investigating the other two directions?

[daanx]

Ah great to hear that. Mmm, not sure why though :-) There must be some interaction where some type of object is allocated by a custom allocator in Qt (say, the pool allocator for the string), but then later it is delete'd using the mimalloc allocator (or the other way around). I believe the mimalloc-new-delete.h overloads are carefully written to only overload the global new/delete so not sure how this happens though...

ps. If you can reproduce reliably, can you try out removing parts of mimalloc-new-delete.h and see which ones are causing the crash? Like:

• remove the "nothrow" overloads?
• remove the array overloads?
• remove the aligned overloads?
• remove the delete overloads that take a size?

not sure.

[drussel]

All of those options still fail (although the exact failure mode does vary a bit). Sorry, I'm actually on mimalloc 1.6.3 (and was on 1.6.1).

I'll see if I can create a simple test case.

[daanx]

Thanks! a small test case would be great if you can manage that. I wonder if we "miss" a new overload that Qt defines, and than one of our delete overloads gets called... or perhaps both Qt and mimalloc try to override the same new/delete?

[drussel]

mimalloc-qt.zip

Has dlls, libs and a trivial main that crashes when the target_compile_definitions(Kaboom PUBLIC OVERRIDE_NEW_DELETE) line in the CMakeLists.txt activating the include of the override header in mimalloc.cpp.

Hope that helps. Thanks very much.

[drussel]

So it I rebuild qt dlls with mimalloc linked in (by including the new-delete override header in qstring.cpp and linking in the libs) the crash goes away.

[kreisbi]

Hi there, I am facing a similar issue as drussel did. I am trying to compile a app with visual studio 2015 (!) that includes mimalloc 2.0.6 as well as Qt 5.9.1. I have tried several ways of including mimalloc and want to use the dynamic overriding. The test projects' main (mimalloc-override-test) runs just fine albeit the debug log showing errors.

But when I am starting my Qt app, I am trying to perform the strdup_test() in my main, I am getting errors.

#include "cqnet2view.h"
#include <stdio.h>
#include <QtWidgets/QApplication>

#include <QtCore/QString>
#include <QtCore/QDateTime>
#include <QtCore/QTextStream>

#include "cfehler.hxx"
#include "skinit.cxx"
#include <mimalloc.h>
#include <new>
#ifdef _WIN32
#include <mimalloc-new-delete.h>
#endif
#include <thread>
#include <assert.h>
#include <vector>
#include <future>
#include <iostream>
#include <stdlib.h>
#include <stdio.h>
#include <assert.h>
#include <string.h>
#include <stdint.h>

#ifdef _WIN32
#include <Windows.h>
static void msleep(unsigned long msecs) { Sleep(msecs); }
#else
#include <unistd.h>
static void msleep(unsigned long msecs) { usleep(msecs * 1000UL); }
#endif

//...
static void strdup_test();            // issue #445 

//...

int main(int argc, char *argv[])
{
	try {
		//...
		strdup_test();
               //...
		
		mi_stats_print(NULL);
		int i = mi_version();
		skinit();

		QApplication a(argc, argv);
		CQNet2View w;
		w.show();
		return a.exec();
	}
	catch (...) {
		return 1;
	}
} 

Also, when I start the actual Qt windows (by getting rid of the tests), the debugger fails at some point with a similar error.

Sorry for the German error messages. If you don't understand them, I will clarify.

If I am starting the application with MI_MALLOC_VERBOSE=3, it fails here:

Is there anything I can do to fix this issue without compiling the Qt dlls by myself?

Any help would be appreciated.

[res2k]

If I were to debug this, my first guess would be that it's a malloc/free mismatch. Perhaps step into strdup_test() and check which allocation functions are actually called.

[kreisbi]

Those are the funktion calls in strdup_test() Unfortunately, the debugger won't let me jump into _strdup("hello\n"); or _dupenv_s(&buf, &len, "MIMALLOC_VERBOSE");, but only to mi_free(buf); and mi_free(s);. The latter are directed to the function void mi_free(void* p) mi_attr_noexcept in alloc.c. Does that help?

[kreisbi]

Okay, I am getting closer.. It seems like my Qt app builds a std::string out of a QString by calling .toStdString(). When this string is deallocated, the error occurs.

[daanx]

Hi, it looks indeed like a pointer mismatch -- which probably means the dynamic override is not working as it should. Can you try to build mimalloc in debug mode? (ie. with -DMI_DEBUG=3). Then run your program with the environment variable MIMALLOC_VERBOSE=1 and you can see if mimalloc was actually redirected. (it should say in the initial output on the console "malloc is redirected").

If this is is not the case, run the bin/minject --list <myexe> and see if the mimalloc-redirect.dll is high up on the import list; best is right after kernel32. (if it is not, try bin/minject <myexe> and run the new <myexe>-mi executable that is written to see if that solves the trouble.)

Let me know how it goes.

[kreisbi]

Sorry for having to ask noob questions, but I am not sure whether I build the debug version correctly. I tried setting "additional build options" in VS to both -DMI_DEBUG=3 or /DMI_DEBUG:3 (cause I wasn't sure abou the right syntax). Building went fine in both cases and I set MIMALLOC_VERBOSE=1 in the environmet.

When starting the app (test-override or my own net2view), I get this error:

Minject gets me this (own app): and this (test-override):

[kreisbi]

Push :)

[kreisbi]

Dear all, after doing some further research and testing, I finally figured out a way to compile Qt with mimalloc and my bugs disappeard. Still, I'd be eager to learn why this step is necessary. When reading the documentation of mimalloc I thought the two dlls (mimalloc-override and mimalloc-redirect) would take care of any other code's/dll's allocators, meaning that they would automatically redirect their allocations/deallocations to mimalloc. As this does not seem to be the case, some clarification would be appreciated. Best regards.

[nickdademo]

@kreisbi

Did you only get issues in Debug builds? Or Release as well?

[amantia]

I have the same issue and I think it happens in all kind of builds, but depending what you do it might or might not end in a crash. With our (much more complex app) it was working fine on Windows 10/11 until we saw that it is crashing on Windows Server 2012 in RelWithDebInfo builds. After many tries I came to a similar conclusion as here (too bad I didn't notice earlier): strings, vectors allocated in Qt library are allocated without using mi-malloc's new , while they are deleted using mi-malloc's delete. The attached example shows this. Start to debug and put breakpoints on the mimalloc operators. Set the MIMALLOC_VERBOSE=3 env var too.

There are several cases:

• if mimalloc.dll was injected (see the cmake code doing it), the crash is more obvious and will be reported

mimalloc: warning: mi_free_size: pointer might not point to a valid heap region: 0x0200000D0E20
(this may still be a valid very large allocation (over 64MiB))
mimalloc: error: mi_free_size: pointer does not point to a valid heap space: 0x0200000D0E20
mimalloc: warning: mi_free: pointer might not point to a valid heap region: 0x0200000D0E20
(this may still be a valid very large allocation (over 64MiB))
mimalloc: error: mi_free: pointer does not point to a valid heap space: 0x0200000D0E20

in this case we have

original imported modules (9):
   0: mimalloc-redirect.dll
   1: mimalloc.dll
   2: mimalloc-debug.dll
   3: Qt6Cored.dll
   4: MSVCP140D.dll
   5: VCRUNTIME140D.dll
   6: VCRUNTIME140_1D.dll
   7: ucrtbased.dll
   8: KERNEL32.dll

If you debug into the deletion, you get this in mi_free_size: available = 0 size = 16

• if mimaloc.dll was not injected at least on windows 11 it doesn't crash and no error is reported. In this case we have

original imported modules (7):
   0: mimalloc-debug.dll
   1: Qt6Cored.dll
   2: MSVCP140D.dll
   3: VCRUNTIME140D.dll
   4: VCRUNTIME140_1D.dll
   5: ucrtbased.dll
   6: KERNEL32.dll

If you debug you will see though that the same happens, operator new is called once, operator delete is called twice for std::strings. In this case though available = 16, size =16 !

In RelWithDebInfo on Windows 11 I couldn't reproduce, but the small testcase might suffer from optimizations (as it shows in xmemory allocation goes through some _Fake_proxy_ptr_impl , the code path is definitely different. In this case minject gives this:

original imported modules (13):
   0: mimalloc-redirect.dll
   1: mimalloc.dll
   2: Qt6Core.dll
   3: MSVCP140.dll
   4: VCRUNTIME140.dll
   5: VCRUNTIME140_1.dll
   6: api-ms-win-crt-runtime-l1-1-0.dll
   7: api-ms-win-crt-heap-l1-1-0.dll
   8: api-ms-win-crt-string-l1-1-0.dll
   9: api-ms-win-crt-math-l1-1-0.dll
  10: api-ms-win-crt-stdio-l1-1-0.dll
  11: api-ms-win-crt-locale-l1-1-0.dll
  12: KERNEL32.dll

Obviously removing #include <mimalloc-new-delete.h> helps though, but doesn't it essentially turn off the mimalloc allocator for all C++ code?

Is there a solution aside of patching and rebuilding Qt?

mimalloc-qt.zip

For testing I used Qt 6.8.3 and the latest MSVC/Visual Studio 2022. Note that to debug into Qt as it is supplied by the Qt Company, you need an older toolkit for MSVC too, 14.39.33519

[drussel]

A few notes:

• we had solved it by patching qt and rebuilding
• we then ran into issues with SentinelOne where by attaching to the program on startup it would apparently perturb something about DLL loading order and end up with crashes on free when mimalloc was used. The only solution for this we found was to completely exclude the executable from monitoring.
• ttb malloc has a path where it catches the structured exception from free and then falls back on the system allocator, presumably to handle exactly this case
• due to issues with tbb malloc and recent visual studio redistributables we are testing out the Windows Segment allocator and so far the performance looks to be OK (with the exception of under load in windows docker containers)

Message ID: @.***>

[amantia]

If all the allocations cannot be redirected I think the proper way to deal with this is to make sure mimalloc doesn't try to deallocate memory that was not allocated by itself, and calls the default delete/free method. I think this is the same what you describe above.