libvips: conversion to pyramidal files fails

[bkocev]

Dear all,

when using the C api of libvips/8.15.1, the conversion to pyramidal files does not work at all using the function:

vips_tiffsave( inputImage, filename, "pyramid", true, "tile", true, "compression", VIPS_FOREIGN_TIFF_COMPRESSION_LZW, "bigtiff", true, NULL )

while when using libvips/8.14.2 it seems to work for "smaller" images (e.g., 13811 X 6122). But for "larger" images (e.g., 122880 X 110592) images, the conversion to a pyramidal file fails with this error:

VIPS:ERROR:../src/libvips/iofuncs/target.c:482:vips_target_flush: assertion failed: (target->write_point >= 0) Bail out! VIPS:ERROR:../src/libvips/iofuncs/target.c:482:vips_target_flush: assertion failed: (target->write_point >= 0)

We suspect there is an issue with the conan build. Please see the discussion at the libvips forum: https://github.com/libvips/libvips/discussions/3828

Is it possible to fix this error with the conan build of libvips?

Thank you, Bojan.

[memsharded]

Hi @bkocev

Thanks for your question.

This wouldn't be a web ticket, I am moving this to the conan repo, though it might need to be moved to the conan-center-index repo if this results to be a recipe issue.

[memsharded]

We would need something to reproduce. Could you please share the following?:

• A minimal conanfile with the minimal to reproduce
• The conan install command and output (including the Conan version)
• A minimal CMakeLists.txt + main.cpp
• The cmake exact commands used.

Thanks very much!

[bkocev]

Hi,

I am not sure I can provide with all the files quickly. This is how I configure the libvips:

conan_cmake_configure( libtiff/4.6.0 libvips/8.14.2 libwebp/1.3.2 GENERATORS cmake_find_package IMPORTS "bin, *.dll -> ${PROJECT_BINARY_DIR}/bin/" )

conan --version Conan version 1.62.0

If I manage to prepare all the files you asked for, I will get back to you.

Best, Bojan.

[memsharded]

Some quick feedback about it:

• The cmake_find_package generator is legacy, even in 1.X, and it is recommended to move to the CMakeDeps generator
• The imports approach is legacy, it is recommended to move to explicit copy() in the generate() method, though for dlls, it might be recommended to use environment generators like VirtualRunEnv
• Conan 2.0 has been out for 1 year now, it is very strongly recommended to move to it as soon as possible, packages in ConanCenter like libvips might stop being compatible with Conan 1.X some time in the following months

Looking forward to getting your feedback and files to reproduce.

[bkocev]

Hi,

I finally managed to prepare a conan project: https://drive.google.com/file/d/1O44xXwru8to9LgXVWLRMToOlMzVvjJLB/view?usp=drive_link. I was not able to upload the zip because it is too big because of the image therein.

I followed the following example https://docs.conan.io/2/tutorial/consuming_packages/build_simple_cmake_project.html. I tested the program on Ubuntu. The command conan profile detect --force created a default profile in Release mode. After doing all the commands and building the project in Release and then running it I got a segmentation fault.

However, after editing the conan default profile to be in Debug mode and then repeating all the commands and building the project in Debug mode, I was able to create a correct pyramidal file. Then repeating it again in Release mode, I was able to create a correct pyramidal file. So, I am guessing there is some issue with the release version of the libvips library in conan, but I am not sure.

Maybe, you can have a look at the problem using the files I provided above and try to make better sense of the potential problem.

Thank you and kind regards, Bojan.

[bkocev]

P.S. The original multi-resolution data is available in the demo data: https://openslide.org/demo/ Hamamatsu NDPI -> OS-1.

[memsharded]

Thanks for sharing your feedback.

I have tried to check

I finally managed to prepare a conan project: https://drive.google.com/file/d/1O44xXwru8to9LgXVWLRMToOlMzVvjJLB/view?usp=drive_link. I was not able to upload the zip because it is too big because of the image therein.,

But it seems that file in google drive is private, and need access. We usually recommend to put things in a Github repo instead, it is easy and very convenient, because we can do Pull Requests, or do code reviews directly here, so much better than in a google drive zip.

[bkocev]

Sorry about this, what about this link https://drive.google.com/file/d/1O44xXwru8to9LgXVWLRMToOlMzVvjJLB/view?usp=sharing

I will see if I can put it later on github.

[memsharded]

Nop, sorry, too big (almost 1Gb) to be scanned, so download now is public but blocked by security. If you can put it later in github, let me know.

[bkocev]

Sorry, here is the github repo with the code https://github.com/bkocev/conan-libvips-problem. And the image file is available here : https://drive.google.com/file/d/1lkezyFyQn14Khwq2kbDIjPB5RF1-3Hbq/view?usp=sharing

I was not able to upload the image file on github as it is too big.

[bkocev]

Hi, were you able to get access to the code and the image file?

Thank you, Bojan.

[memsharded]

Thanks for the ping.

I have reproduced the steps in https://github.com/bkocev/conan-libvips-problem/blob/main/run_example.sh in the conanio/gcc11 docker image, using Conan 2.0, and it worked fine:

conan@61aba5f145c1:~/conanws/kk/conan-libvips-problem/build$ ./PyramidalImagesConverter ../hamamatsu-os-1-as-singleres.tiff
Successfully loaded the image.

VIPS works!!!

I'll try with 1.62 too

[memsharded]

It seems the project you put together is for 2.0 only, it is not working with 1.62 (it would need a different [layout], for example, and different commands). So it seems that you also did it with Conan 2.0.

It would be great if you can test it in the docker image I suggest above, to see if you can reproduce it with it.

[bkocev]

HI,

I am using Conan 1.62.0 on Windows, but I was not able to prepare an example project for you under Windows and hence I provided one with Conan 2.0 under Linux. I tried it on my native Ubuntu installation, i.e., not in a docker container. And, I was able to reproduce the problem as I described.

Best, Bojan.

[memsharded]

I have tried in my Ubuntu 22.04 (WSL) not docker, and it works too:

 ./PyramidalImagesConverter ../hamamatsu-os-1-as-singleres.tiff
Successfully loaded the image.
VIPS works!!!

It could be something related to having an older glibc in your specific ubuntu (but I wouldn't know in Windows, this shouldn't be the case). What ubuntu are you using? If using an older one, it is worth trying:

• Make sure to build all dependencies from source, just in case --build=*
• Try in a more modern Ubuntu

[bkocev]

Hi,

I am using Ubuntu 22.04.3 LTS. I could reproduce the problem only once on Ubuntu, as described above. However, after editing the conan default profile to be in Debug mode and then repeating all the commands and building the project in Debug mode, I was able to create a correct pyramidal file. Then repeating it again in Release mode, I was able to create a correct pyramidal file. So, my guess was that there is some issue with the release version of the libvips library in conan, but I am not sure. At the moment, the problem is that you cannot reproduce the issue I have and I do not know what to suggest. It would be nice if you can try it on Windows too. I could not get the exampleto work on Windows though.

Best, Bojan.

[memsharded]

I have done the steps in Windows, and it seems to run:

(conanpip2_310) λ PyramidalImagesConverter ../hamamatsu-os-1-as-singleres.tiff
Successfully loaded the image.

And I can see the file hamamatsu-os-1-as-singleres.tiff_multires which is double the size.

It is true that the message printf("VIPS works!!!\n"); is not printed. It would seem that the program is not finalizing correctly.

This kind of runtime error can be quite challenging to understand where is coming from 😞

Maybe that could be a library issue? From the discussion in https://github.com/libvips/libvips/discussions/3828 it is not fully explicit that they are trying that in Windows?

[bdal-jsinge]

Hi, this is Jonas, a colleague of Bojan.

I managed yesterday to build his example in release mode with MSVC 2022 on Windows. I do get an access violation after the program runs for some time. I tried three times, it always stops when the result file reaches exactly 2 GB file size. @jcupitt does this ring a bell to you? Maybe some configuration setting in the dependencies? I'll try to get a meaningful callstack and get back to you.

[jcupitt]

Oh, interesting @bdal-jsinge!

TIFF pyramid generation runs in two main phases: it renders every level in parallel to a set of separate TIFF files, then in a second pass gathers the $n TIFFs together into one pyramid image. Perhaps the gather step is not working correctly for >2GB files? That's the size limit for non-bigtiff TIFFs. I'll have a look.

Windows is a bit more complicated -- they have a separate API for large files which you need to enable at compile time. That could be missing on the windows build you are using.

[jcupitt]

I tried on ubuntu:

$ vips tiffsave C2023000589S1-0-1_743eee76-f53b-aca2-4da5-fa97a44d9933_125232.svs x.tif --tile --pyramid --compression jpeg --bigtiff --vips-progress
vips temp-5: 211213 x 87581 pixels, 32 threads, 128 x 128 tiles, 640 lines in buffer
vips temp-5: done in 217s          
$ ls -l x.tif
-rw-r--r-- 1 john john 2610298042 Feb 17 15:42 x.tif

I examined the TIFF and it seems fine -- I could view every pixel in every level. I tried without the --bigtiff flag and that worked too, I suppose libtiff have improved their handling of files in the 2gb to 4gb range since I'm pretty sure this used to fail.

My guess would be that conan is not building libtiff on windows with the large file backend enabled (but it's just a guess).

A simple test for this might be:

$ vips black x.tif 100000 100000 --vips-progress
vips temp-1: 100000 x 100000 pixels, 32 threads, 100000 x 1 tiles, 640 lines in buffer
vips temp-1: done in 11.7s          
$ file x.tif
x.tif: Big TIFF image data, little-endian
$ vips avg x.tif --vips-progress
vips temp-1: 100000 x 100000 pixels, 32 threads, 100000 x 1 tiles, 640 lines in buffer
vips temp-1: done in 43.6s          
0.000000

If that fails, then I think it's likely to be a misconfiguration of libtiff.

[bdal-jsinge]

Yes, the test you suggested is failing for vips.exe from the conan windows package, already when trying to write the image:

C:\Users\jonas>C:\Users\jonas\.conan2\p\b\libvidf931674c7877\b\build-debug\tools\vips.exe black x.tif 100000 100000 --vips-progress vips.exe temp-0: 100000 x 100000 pixels, 20 threads, 100000 x 16 tiles, 384 lines in buffer vips.exe temp-0: done in 0.876s TIFFAppendToStrip: Maximum TIFF file size exceeded wbuffer_write: write failed memory: high-water mark 85.45 MB error buffer: TIFFAppendToStrip: Maximum TIFF file size exceeded wbuffer_write: write failed

I looked a little bit around for configuration options on the libtiff library, but it's not clear yet to me how to enable files larger than 2 GB for msvc builds. Found some documentation here but nothing specific how to enable large file support: https://libtiff.gitlab.io/libtiff/build.html#id2 Here's a configuration file that seems to enable it only for mingw or unix platforms: https://gitlab.com/libtiff/libtiff/-/blob/master/cmake/LargeFileSupport.cmake?ref_type=heads

I also don't see anything specific in the vcpkg recipe. But I also don't know if it would show the same problem or not: https://github.com/microsoft/vcpkg/blob/master/ports/tiff/portfile.cmake

[bdal-jsinge]

Regarding the crash I observed: I guess it's a subsequent problem after the tiff library write failure. Can't tell whether it's worth fixing it, but here's some more info:

The crash happens due to nullpointer access in src\libvips\iofuncs\target.c:459, as VipsTargetClass *class is a nullpointer. It happens when releasing resources after the vips_sink_disc call fails src\libvips\foreign\vips2tiff.c:2285

Callstack looks like this:

PyramidalImagesConverter.exe!vips_target_write_unbuffered(_VipsTarget * target, const void * data, unsigned __int64 length) Line 459 at packagePath\src\libvips\iofuncs\target.c(459) PyramidalImagesConverter.exe!vips_target_flush(_VipsTarget * target) Line 488 at packagePath\src\libvips\iofuncs\target.c(488) PyramidalImagesConverter.exe!vips_target_seek(_VipsTarget * target, long position, int whence) Line 584 at packagePath\src\libvips\iofuncs\target.c(584) PyramidalImagesConverter.exe!openout_target_seek(void * st, unsigned __int64 offset, int whence) Line 228 at packagePath\src\libvips\foreign\tiff.c(228) PyramidalImagesConverter.exe!TIFFLinkDirectory(tiff * tif) Line 2956 at tiffPackagePath\src\libtiff\tif_dirwrite.c(2956) PyramidalImagesConverter.exe!TIFFWriteDirectorySec(tiff * tif, int isimage, int imagedone, unsigned __int64 * pdiroff) Line 1098 at tiffPackagePath\src\libtiff\tif_dirwrite.c(1098) PyramidalImagesConverter.exe!TIFFWriteDirectory(tiff * tif) Line 238 at tiffPackagePath\src\libtiff\tif_dirwrite.c(238) PyramidalImagesConverter.exe!TIFFRewriteDirectory(tiff * tif) Line 317 at tiffPackagePath\src\libtiff\tif_dirwrite.c(317) PyramidalImagesConverter.exe!TIFFFlush(tiff * tif) Line 50 at tiffPackagePath\src\libtiff\tif_flush.c(50) PyramidalImagesConverter.exe!TIFFCleanup(tiff * tif) Line 51 at tiffPackagePath\src\libtiff\tif_close.c(51) PyramidalImagesConverter.exe!TIFFClose(tiff * tif) Line 156 at tiffPackagePath\src\libtiff\tif_close.c(156) PyramidalImagesConverter.exe!layer_free(_Layer * layer) Line 946 at packagePath\src\libvips\foreign\vips2tiff.c(946) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 961 at packagePath\src\libvips\foreign\vips2tiff.c(961) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!layer_free_all(_Layer * layer) Line 960 at packagePath\src\libvips\foreign\vips2tiff.c(960) PyramidalImagesConverter.exe!wtiff_free(_Wtiff * wtiff) Line 975 at packagePath\src\libvips\foreign\vips2tiff.c(975) PyramidalImagesConverter.exe!vips__tiff_write_target(_VipsImage * input, _VipsTarget * target, VipsForeignTiffCompression compression, int Q, VipsForeignTiffPredictor predictor, const char * profile, int tile, int tile_width, int tile_height, int pyramid, int bitdepth, int miniswhite, VipsForeignTiffResunit resunit, double xres, double yres, int bigtiff, int rgbjpeg, int properties, int strip, VipsRegionShrink region_shrink, int level, int lossless, VipsForeignDzDepth depth, int subifd, int premultiply, int page_height) Line 2286 at packagePath\src\libvips\foreign\vips2tiff.c(2286) PyramidalImagesConverter.exe!vips_foreign_save_tiff_build(_VipsObject * object) Line 213 at packagePath\src\libvips\foreign\tiffsave.c(213) PyramidalImagesConverter.exe!vips_foreign_save_tiff_file_build(_VipsObject * object) Line 519 at packagePath\src\libvips\foreign\tiffsave.c(519) PyramidalImagesConverter.exe!vips_object_build(_VipsObject * object) Line 367 at packagePath\src\libvips\iofuncs\object.c(367) PyramidalImagesConverter.exe!vips_cache_operation_buildp(_VipsOperation * * operation) Line 912 at packagePath\src\libvips\iofuncs\cache.c(912) PyramidalImagesConverter.exe!vips_call_required_optional(_VipsOperation * * operation, char * required, char * optional) Line 913 at packagePath\src\libvips\iofuncs\operation.c(913) PyramidalImagesConverter.exe!vips_call_by_name(const char * operation_name, const char * option_string, char * required, char * optional) Line 953 at packagePath\src\libvips\iofuncs\operation.c(953) PyramidalImagesConverter.exe!vips_call_split(const char * operation_name, char * optional, ...) Line 1057 at packagePath\src\libvips\iofuncs\operation.c(1057) PyramidalImagesConverter.exe!vips_tiffsave(_VipsImage * in, const char * filename, ...) Line 748 at packagePath\src\libvips\foreign\tiffsave.c(748) PyramidalImagesConverter.exe!main(int argc, char * * argv) Line 33 at C:_DEV\conan-libvips-problem\src\main.c(33) [Inline Frame] PyramidalImagesConverter.exe!invoke_main() Line 78 at D:\a_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl(78) PyramidalImagesConverter.exe!__scrt_common_main_seh() Line 288 at D:\a_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl(288) kernel32.dll!BaseThreadInitThunk() ntdll.dll!RtlUserThreadStart()

[jcupitt]

libvips is opening the output TIFF in w8 mode, ie. bigtiff write. For some reason the conan libtiff build does not seem to support this (I think?) hence the failure.

It looks like libtiff last revised all this back in 4.0.5:

https://github.com/libsdl-org/libtiff/blob/master/doc/releases/v4.0.5.rst

Does any of that look relevant? Could conan be building for a very old MSVC runtime?

You could try to check that conan's tiffcp has bigtiff working correctly, I see:

$ tiffcp -8 k2.tif x.tif
$ file x.tif
x.tif: Big TIFF image data, little-endian
$ file k2.tif
k2.tif: TIFF image data, little-endian, direntries=18, height=2048, bps=23318, compression=JPEG, PhotometricInterpretation=YCbCr, orientation=upper-left, width=1450

[bdal-jsinge]

I think the big tiff format extension itself is enabled in the conan libtiff artifact.

By default, tiff-tools are not built, but I modified the recipe and have built it locally to get the tiffcp tool.

With a small tif file I get the bigtiff file from tiffcp:

tiffcp -8 -m 0 C:\_DEV\conan-libvips-problem\example.tiff tiffcp-900mb.tiff
file ~/tiffcp-900mb.tiff
/c/Users/jonas/tiffcp-900mb.tiff: Big TIFF image data, little-endian

and for a file of 20 GB size as well:

tiffcp -8 -m 0 C:\_DEV\conan-libvips-problem\in-20gb.tif tiffcp-20gb.tif
$ file ~/tiffcp-20gb.tif
/c/Users/jonas/tiffcp-20gb.tif: Big TIFF image data, little-endian
$ ls -lh ~/tiffcp-20gb.tif
-rw-r--r-- 1 Jonas 1049089 21G Feb 19 14:33 /c/Users/jonas/tiffcp-20gb.tif

Note that "file" is not available on windows natively, but I used the one from git bash.

[jcupitt]

How strange! Could you try:

vips.exe black x.tif[bigtiff] 100000 100000 --vips-progress

[jcupitt]

I fired up a windows VM and tried with the vips.exe we distribute:

F:\vips-dev-8.15\bin>vips.exe black x.tif[bigtiff] 100000 100000 --vips-progress
vips.exe temp-1: 100000 x 100000 pixels, 8 threads, 100000 x 1 tiles, 256 lines in buffer
vips.exe temp-1: done in 3.36s
x.tif: write error
windows error: The printer is out of paper.

TIFFAppendToStrip: Write error at scanline 23552
wbuffer_write: write failed

So I suppose the libvips build system has this issue with libtiff configuration too. @kleisauke, could I ping you on this?

[kleisauke]

I was unable to reproduce this on Windows 11 using the prebuilt binaries available at https://github.com/libvips/build-win64-mxe. Tested both 32-bit and 64-bit versions.

PS> vips black x.tif[bigtiff] 100000 100000 --vips-progress
vips.exe temp-1: 100000 x 100000 pixels, 24 threads, 100000 x 1 tiles, 512 lines in buffer
vips.exe temp-1: done in 2.31s
PS> vipsheader x.tif
x.tif: 100000x100000 uchar, 1 band, b-w, tiffload
PS> "Size in bytes: {0}" -f (Get-Item "x.tif").Length
Size in bytes: 10000009716

[jcupitt]

Ooof, so sorry, my disk was filling up and that was stopping the write! I suppose the error codes are not being mapped into strings correctly -- it should be disk full, not printer out of paper.

Yes, it's all working for me too with the official libvips binaries, thank you @kleisauke

[jcupitt]

You can see how libvips is building libtiff here:

https://github.com/libvips/build-win64-mxe/blob/master/build/overrides.mk#L663-L683

That's using configure and I think conan is using cmake, so you'd need to investigate the cmake build script.

[kleisauke]

I suppose the error codes are not being mapped into strings correctly -- it should be disk full, not printer out of paper.

Indeed, that error message looks weird. It looks like the conversion takes place here: https://github.com/libvips/libvips/blob/v8.15.1/libvips/iofuncs/error.c#L301-L311

Perhaps we should use the portable and thread-safe g_strerror() function there? That one uses strerror_s() on Windows: https://github.com/GNOME/glib/blob/2.79.2/glib/gstrfuncs.c#L1305