1.5 - Large memory comsumption with large textures (eg. 4098x4098)

[ffyhlkain]

I'm trying to drag & drop a 4098x4098 texture into the "Textures" folder, but the application crashes then. Haven't seen a texture size limit, but I guess this is unintended.

Using latest Windows 1.5 portable version, but the same happens with 1.4. Smaller texture are working fine.

[azagaya]

I confirm it happens on some devices, i'll look into it. It only seems to affect windows, cause i've tried on multiple Linux devices and didn't happen.

[azagaya]

Apparently there is a problem with memory allocation in opencv for windows... i wish this can be solved by my side.

OpenCV Error: Insufficient memory (Failed to allocate 603979776 bytes) in OutOfMemoryError, file C:\opencv\sources\modules\core\src\alloc.cpp, line 52 OpenCV Error: Assertion failed (u != 0) in create, file C:\opencv\sources\modules\core\src\matrix.cpp, line 433 terminate called after throwing an instance of 'cv::Exception' what(): C:\opencv\sources\modules\core\src\matrix.cpp:433: error: (-215) u != 0 in function create

[ffyhlkain]

When I observe the memory consumption of the application when loading the image, memory usage is increased step by step, so I guess the crash happens when the app is creating the different textures (normals, parallax,...). So I would suggest to check if a specific step is causing the crash as some of the generations seem to work fine. Additionally I've noticed that the application is only 32 bit. If you can, you should create a 64 bit application (or generate both if it's not too much hassle), as this will also help with memory allocation.

Sadly my C++ skills are a bit rusty, so I haven't tried out the changes myself (yet) :)

[azagaya]

Hi!

So I would suggest to check if a specific step is...

Yes, thats the way i should try.

Additionally I've noticed that the application is only 32 bit.

I've also tried, but opencv with Qt for 64 bit did not correctly compile for me no matter what i've tried. I should try again. Anyways, if my math doesn't betray me, memory should be enough.. i think 32bit can allocate almost 4gb.. I'll start by testing each step to see if something is trying to use more memory than intended

[Gnumaru]

I've also tried, but opencv with Qt for 64 bit did not correctly compile for me no matter what i've tried.

I also could not compile opencv 3.2.0 on windows with mingw 64 bit. But I just tried one more time and managed to make it work.

I'm using windows 10 64 bit and I used the mingw 7.3.0 distribution that comes with Qt. But for it to work I had make sure I removed mingw 32 bit from the system path and added mingw 64. I also had to uncheck BUILD_opencv_python3 (wich I already had to uncheck to make it compile the 32 version).

I had some path related issues before when compiling opencv 32 bit on different machines and solved it "reseting" the variable. The default value for the windows path on a fresh windows 10 install is the one below:

C:\Windows\system32; C:\Windows; C:\Windows\System32\Wbem; C:\Windows\System32\WindowsPowerShell\v1.0\; C:\Windows\System32\OpenSSH\

O course, after reseting the path variable you will need to add the directory path to the mingw version you are using

Unfortunately I didn't find any way to reset a system variable but by overwriting it with another value.

[azagaya]

Great! i was going to try the same thing after work. These days i'm really busy on my day job. What Qt version you used?

And could you test if Laigter is also crashing when opening that big image in 64 bit version?

[Gnumaru]

I'm using Qt 5.13.0

Yeah, the 64 bit build loads the 4096x4096 texture. The application freezes for a brief moment, cpu usage for the proccess goes to 16%, ram usage goes to 1700MB and then the texture loads. The preview on the canvas seems to work correctly and the export also seems to work correctly.

I also tried with an 8192x8192 texture. The ram comsumption goes to 4945MB and the app does not crash, but neiter the canvas shows the texture nor the textures list updates with the texture path.

[Gnumaru]

And a 16384x16384 texture makes the app crash

[azagaya]

Ok, clearly there is something that is comsuming too much ram. Perhaps we should try in the future to cache the images once processed or something.

Anyways, for 1.6, we could just release the 64 bit version so the issue affects less users, and later for future version try to implement a better way.

Nevertheless, i think it's not common to try to automatically generate maps for that big images.. i may be wrong.

Thanks a lot for your help!

[azagaya]

I managed to compile it for 64 bit, didn't have reset path or variables, just add new mingw to path variable. The previous time i tried, i used Qt 5.12 and an unofficial mingw64 for Qt (because it wasn't official till 5.13 i think), so now larger images may be opened.

However, this doesn't solve the issue, cause images are consuming a lot more memory than they should... This may be caused because i have multiple copies of each image in memory, to simplify things... so i'll have to start optimizing that.

@ffyhlkain would you describe me your use case for this kind of images? Perhaps i find a better solution. And do you prefer portable or installer? Cause i'm going to send you the binaries so you can try, if it's ok with you!

Thanks a lot to everybody!

[ffyhlkain]

Great work guys and sure I can. Use case is that I create texture atlases for my characters, which need to be squared to be able to be compressed (PVRTC for example) for mobile. For simplicity I create first the atlas (automatically), then import the texture and generate the normal texture for it. (If I would do it the other way around, I would have to rely on that the same character sprites are placed all the same over both textures, which I wouldn't bet. As I mentioned, it's an automated process).

Textures shouldn't get larger than 4098x4098 though.

I'd take the portable version, but I'm also fine with the installer ^^

[azagaya]

Ah ok, that's a good example of use case indeed. I've seen other people first doing normals, and then making the atlases, but now that i think, that would duplicate the effort, as you would need to do atlases twice and also make normals for all individual sprite. However, may be it would be a nicer approach to be able to parse the atlases and load images one by one, and may be cache them... but for the future.