serverless-iiif
serverless-iiif copied to clipboard
samvera
Mostly jp2 images are not opening it is working in cantaloupe.Attaching error screenshot
Image url=https://uat-iiif.newmanstudies.org/iiif/2/TRT%2F01_Franciscan_Christocentrism_page-0003.jp2/full/full/0/default.jpg
I was wondering if this was resolved. I ran into the same problem when using jp2 images.
@ranjeetsinghthirdrock @guillenjs Do you have any jp2 source images you can share with me so that I can try to reproduce and fix this issue? You can either attach them here or email them to [my github username] at gmail.
@ranjeetsinghthirdrock After some debugging with a sample file provided by @guillenjs, I have narrowed the issue down to the JP2 loader in the underlying libvips image processing library. I have submitted a bug report in hopes of getting things working. If either of you would be willing to let me attach any of the files that are giving you trouble to that issue (or even attach them yourselves in comments), it would go a long way to helping resolve the issue.
@mbklein here is another sample image that is not opening https://4slvho3i5lch7kp6zdultsdumq0ihqta.lambda-url.us-east-1.on.aws/iiif/2/TRT%2F01_A8.jp2/full/100,/0/default.jpg
@ranjeetsinghthirdrock Would it be possible to share the source JP2 file for that image? Either by zipping/attaching it here, or by emailing it to mbklein at gmail?
@ranjeetsinghthirdrock @guillenjs The underlying issue is with the OpenJPEG library that provides JPEG 2000 support. It looks like there's going to be a libvips release with a workaround – not an ideal one, as it depends on loading the entire frame instead of just the tiles required to complete the request, but I've tested it on my development server and the lag isn't terrible. But it's the best option available until the upstream issue is addressed.
I've also reached out to the team at Kakadu to ask how their licensing might work given the way our distribution works. It would require downstream users (i.e., you) to obtain a valid license for the Kakadu libraries, but if you've had this working in Cantaloupe, it's possible you've already been using the KakaduNativeProcessor with a license? (If you've been using the Cantaloupe OpenJpegProcessor, you may not even notice the performance hit from the libvips workaround mentioned above.)
@mbklein Thank you for all the work on this front! Is there a projected date when this update may be released so we can start testing on our end?
I'm currently testing with the libvips branch with the fix in it. Once it's stable, the maintainer will have to approve, merge, and release it, and then I can build and push a Lambda layer with support for it. I hope it'll be soon – closer to "a couple weeks" than "a few weeks."
I'm afraid I didn't quite understand the libvips release schedule, so it might be six months or so before the fix is in a release. That said, there's nothing stopping me from building a lambda layer with the pre-release workaround in it, so I'll let you know when it's available.
@ranjeetsinghthirdrock @guillenjs I've just released serverless-iiif v5.1.1 with support for the sharp/libvips/openjpeg workaround for this issue. It uses as-yet unreleased code from libvips and a forked version of sharp that enables the workaround code. I'll update to the released versions as the code make it into upstream releases.
If you have a chance, could you install or upgrade to that version and see if it fixes the problems you're having?
@mbklein, thank you for the update. I will test this out this week and get back to you about it.
@mbklein I have been testing the new v5.1.1 release, and generally, everything seems to be working fine. I did try and find a large file that is 20,000+ pixels and 18.6MB. That is the one file that I seem to be struggling to work correctly. It is not the same issue as before. If I request a smaller size, I get the image returned, but when I request a full-size image, I get an "Internal server error."
So, I am wondering if there is a max pixel width/height that would be ideal to stay under.
Thanks again for the all the help!
It's less about pixel width and more about memory – if you look at the CloudWatch logs for that request you'll probably see something like
REPORT RequestId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Billed Duration: 3000 ms Memory Size: 3008 MB Max Memory Used: 3009 MB
RequestId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Process exited before completing request
If you deployed it using the default parameters, you could try bumping up the IiifLambdaMemory setting. It can go as high as 10240. But since the workaround for the issue we've been focused on involves loading the entire image into memory in one shot (instead of one tile at a time), it's also very likely that very large JPEG 2000 images with the multipart tile layout will eat way more memory than they would otherwise.
Hello, I saw related issue opened on openjpeg tracker - there is an alternative to OpenJPEG and Kakadu that might help
https://github.com/GrokImageCompression/grok https://www.bitsgalore.org/2022/03/30/generating-lossy-access-jp2s-from-lossless-preservation-masters
Grok (an older version) has been integrated into Cantaloupe but I'm not sure about adoption.
Thanks for pointing that out, @boxerab. I've passed that info along to the libvips maintainer; it'll be up to him to decide whether to add support for it.
libvips v8.17.0 has been released with support for the jp2Oneshot option, and the flag to enable it is in the current sharp release candidate. Once sharp is released with the flag, I'll update the lambda layer with the released versions.
@mbklein was the jp2Oneshot option added into the 6.0.1 release? Unfortunately, still seeing the issue if it was added. I can provide a source image to reproduce if it helps with your testing. Thanks in advance.
START RequestId: 66ca7d44-7eb3-4af8-800c-831bd123c648 Version: $LATEST
2025-11-19T15:26:42.100-08:00
2025-11-19T23:26:42.100Z 66ca7d44-7eb3-4af8-800c-831bd123c648 WARN Unable to get dimensions for healthcheckimage.jpx using custom function. Falling back to sharp.metadata().
2025-11-19T15:26:42.429-08:00
2025-11-19T23:26:42.429Z 66ca7d44-7eb3-4af8-800c-831bd123c648 INFO Encountered JP2 tile part index error. Trying oneshot load.
2025-11-19T15:26:42.470-08:00
2025-11-19T23:26:42.470Z 66ca7d44-7eb3-4af8-800c-831bd123c648 WARN Unable to get dimensions for healthcheckimage.jpx using custom function. Falling back to sharp.metadata().
2025-11-19T15:26:42.675-08:00
2025-11-19T23:26:42.675Z 66ca7d44-7eb3-4af8-800c-831bd123c648 ERROR Error: jp2kload_buffer: Invalid tile part index for tile number 1. Got 6, expected 0 jp2kload_buffer: Fail to read the current marker segment (0xff90) jp2kload_buffer: Failed to decode the codestream in the JP2 file at Sharp.toBuffer (/opt/nodejs/node_modules/sharp/lib/output.js:163:17) at /var/task/index.js:44144:32 at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async Processor.withStream (/var/task/index.js:44056:14) at async Processor.iiifImage (/var/task/index.js:44137:20) at async Processor.execute (/var/task/index.js:44160:14) at async executeWithJp2Retry (/var/task/index.js:44506:14) at async handleImageRequest (/var/task/index.js:44523:20) at async /var/task/index.js:44463:16 at async Runtime.handler (/var/task/index.js:44410:20)
2025-11-19T15:26:42.678-08:00
END RequestId: 66ca7d44-7eb3-4af8-800c-831bd123c648
2025-11-19T15:26:42.678-08:00
REPORT RequestId: 66ca7d44-7eb3-4af8-800c-831bd123c648 Duration: 600.45 ms Billed Duration: 601 ms Memory Size: 2048 MB Max Memory Used: 162 MB ```
@cachemeoutside Yes, jp2Oneshot should be working in 6.0.1. A source image would be great.
v6.0.2 released. I missed where sharp had nested all of its format-specific constructor options, so jp2Oneshot became jp2.oneshot.