Add HTJ2K Compressor

[palemieux]

WIP DO NOT MERGE

This patch fulfills my action item from the September 19 TSC meeting.

Introduction

This patch proposes to add support for High-Throughput JPEG 2000 (HTJ2K) compression to OpenEXR -- HTJ2K is JPEG 2000 with the HT block coder standardized in Rec. ITU-T T.814 | ISO/IEC 15444-15. As detailed at Evaluating HTJ2K as a compressor option for OpenEXR, HTJ2K demonstrates significant improvements in both speed and file size over other OpenEXR compressors. Furthermore, HTJ2K is a worldwide standard that benefits from a diversity of implementations, builds on JPEG 2000, which is in broad use in studio workflows, is appropriate for long-term preservation, has both lossy and lossless modes and supports integer and floating point samples. Finally, the unique features of HTJ2K could be powerful additions to high-performance OpenEXR workflows where the full image may not always be viewed at its full resolution.

The patch currently defines four compressors:

• HT: Lossless coding using HTJ2K. Each chunk is the full image frame and the open-source OpenJPH library is used.
• HT256: Lossless coding using HTJ2K. Each chunk is 256 lines and the open-source OpenJPH library is used.
• HTK: Lossless coding using HTJ2K. Each chunk is the full image frame and the commercial KDU library is used.
• HTK256: Lossless coding using HTJ2K. Each chunk is 256 lines and the commercial KDU library is used.

Questions

• can a compressor store configuration information once in a file, so that the same information is available for each chunk to both the compressor and decompressor?

Notes

• HT and HTK Compressors are identical and use the open-source OpenJPH library is the Kakadu SDK is not found
• Ultimately there will be a single lossless HTJ2K Compressor. The HT and HTK Compressors are currently offered to ensure interop, and facilitate comparisons, between OSS and commercial options.
• The full-frame and 256-line chunk options are currently offered to evaluate the impact of sub-frame chunking on coding efficiency performance.

Todo

• add support for 32-bit samples to OpenJPH
• explore adding a progressive decoding API to OpenEXR
• explore tradeoff of chunk-size (currently set to 256) on compression efficiency, single-threaded/multi-threaded throughput and partial decoding (low-resolution and/or region of interest)
  • can chunking can be replaced with full-frame coding and J2K accessibility features? This is more important in lossy coding to avoid edge effects.
  • select a chunk size, i.e., is 256-line a reasonable value?
  • can J2K tiles be used instead of independent codestreams to remove JPEG 2000 codestream header overhead?

[linux-foundation-easycla[bot]]

The committers listed above are authorized under a signed CLA.

• :white_check_mark: login: palemieux / name: Pierre-Anthony Lemieux (fb870b6c8f50ad75215ca5736c20810c4944a86d, 4afc29f668e10e4ef1aa4d2614ad33be516f4816, ebf6d492e36416056477e73d3ed35e1653d40de1, 04fcf2f265415edcfd40817e35891998f53f713e, e73568204de5775bc772fcf1bc709ad2a2f6e24b)

[cary-ilm]

The CI is failing in the validate step, which is attempting to link a test program against the library. The error message indicate the link is failing with unresolved symbols, possibly because that cmake configuration is missing the new dependency.

However, this part of the CI has been rewritten in the PR I just merged. Can you rebase your branch onto the current main branch now? It may still fail, but hopefully it will be easier to resolve then at least.

[palemieux]

Can you rebase your branch onto the current main branch now? It may still fail, but hopefully it will be easier to resolve then at least.

@cary-ilm Done.

[cary-ilm]

The failure in the "Validate" step of the linux build is along the lines of what we discussed in the TSC meeting today, the check to make sure "make install" installs just the right files. This output from the logs appears to indicate that the cmake configuration is causing the openjph library and headers to be installed, when that shouldn't happen, it'll cause problems for downstream projects:

Error: The following files were installed but were not expected:
  bin/ojph_compress
  bin/ojph_expand
  include/OpenEXR/ImfHTCompressor.h
  include/OpenEXR/ImfHTKCompressor.h
  include/openjph/ojph_arch.h
  include/openjph/ojph_arg.h
  include/openjph/ojph_base.h
  include/openjph/ojph_codestream.h
  include/openjph/ojph_defs.h
  include/openjph/ojph_file.h
  include/openjph/ojph_mem.h
  include/openjph/ojph_message.h
  include/openjph/ojph_params.h
  include/openjph/ojph_version.h
  lib64/libopenjph.so
  lib64/libopenjph.so.0.18
  lib64/libopenjph.so.0.18.2
  lib64/pkgconfig/openjph.pc

Try to make sure your cmake "fetch" of openjph mirror as close as possible how the current setup handles libdeflate and Imath. I had a similar issue with them originally but the current setup should handle the the fetch properly.

[palemieux]

Thanks! What is the best way to run that validate step locally?

[palemieux]

@cary-ilm I remember now... I could never fix the following errors:

[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "Iex" which requires target "openjph" that is not in any export set.
[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "IlmThread" which requires target "openjph" that is not in any export set.
[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "OpenEXRCore" which requires target "openjph" that is not in any export set.
[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "OpenEXR" which requires target "openjph" that is not in any export set.
[cmake] CMake Error: install(EXPORT "OpenEXR" ...) includes target "OpenEXRUtil" which requires target "openjph" that is not in any export set.

It seems to impose some requirements on openjph, which I could not find a solution for.

[cary-ilm]

I'm not enough of a cmake expert to immediately know how to resolve this. Since this is a work in progress, you could just disable the CI "validate" step entirely for now, just edit the workflow file on your branch and add a "if: false" line, or just delete it. We'll need it resolved eventually before merging, but at least that will allow you to get on with other priorities.

I'm happy to help resolve this, but I won't have much time to look into it until mid-January.

[peterhillman]

Regarding exrperf and exrconv: OpenEXR now provides exrmetrics for performance analysis. exrmetrics can also do compression conversion (though that's not an obvious use from the name) so can be used instead of exrconv. It looks like exrmetrics has more complete support for different channel and part types, but there are a couple of features that exrperf and exrconv offer that are missing from exrmetrics. Perhaps it would be better to add those as options to exrmetrics, probably as a separate PR, instead of adding new binaries?

[palemieux]

Perhaps it would be better to add those as options to exrmetrics, probably as a separate PR, instead of adding new binaries?

Yes. exrperf and exrconv will be removed before this is merged.

[palemieux]

@cary-ilm I believe this is ready for review and discussion at the upcoming TSC call.

[kmilos]

Please either make this optional, and/or also support building w/ system OpenJPH, like libdeflate. Bundling a (unreleased) feature branch/tag is not an ideal solution for long-term distribution...

[palemieux]

@kmilos I agree that referencing an unreleased branch/tag of a dependency is not ideal. I expect a release of OpenJPH to be referenced by the time this PR is merged.

[kmilos]

Ok, makes sense, thanks @palemieux In the meantime I still think it would also be good to prepare for the detection for the system library, like it is done for libdeflate...

[palemieux]

In the meantime I still think it would also be good to prepare for the detection for the system library, like it is done for libdeflate...

You mean use the system library if available (through find_package), and fall back to the GitHub dependency if the system library is not available?

[kmilos]

You mean use the system library if available (through find_package), and fall back to the GitHub dependency if the system library is not available?

Exactly, this is how it is already done for libdeflate (one can skip the CONFIG detection branch as OpenJPH doesn't ship a CMake config, only a pkgconf one). And Imath as well. In addition there are options to force use of the "internal" (fetched from GitHub) ones...

[palemieux]

Exactly, this is how it is already done for libdeflate (one can skip the CONFIG detection branch as OpenJPH doesn't ship a CMake config, only a pkgconf one). In addition there is an option to force use of the "internal" (fetched from GitHub) one...

Ok. No objection from me. Is that regularly tested?

[TodicaIonut]

setting HT Compression Quality Level float. visually lossless setting Compression Level 0.0f, quantizing lossy. floating point base-error value. @palemieux

[cary-ilm]

@palemieux, now that we're getting close with this, we do need to resolve the DCO and CLA, note those checks have been failing. The fix for the other failing checks should get merged to main shortly.

For the contributor license agreement you'll need to sign the form through the red "NOT COVERED" link above.

For the "Digital Certificate of Origin", each commit needs to be "signed" via the -s argument to git commit. This adds the Signed-off-by: <email> line to the commit message. You'll need to do this for every commit. You can do this retroactively to the existing commits via git commit -s --amend --no-edit. You'll then have to git push --force to overwrite the existing commits.

You could also squash everything into a single signed commit, since the history of these commits will get lost when we merge it anyway, since we generally "squash-and-merge" instead of "rebase-and-merge".

For the feature branch, we'll also need this PR to merge to the feature branch name, not to main as it stands now. Should we call that AcademySoftwareFoundation/htj2k?

I look this over in the next few days to see if anything further stands out.

[palemieux]

@cary-ilm CLA is executed, DCO is done.

Ok with AcademySoftwareFoundation/htj2k. Can you create the branch and I will change the target branch of this PR and rebase the commit.

[cary-ilm]

@palemieux I created the branch and called it htj2k-beta.

[palemieux]

@cary-ilm The white paper is in error :(. The official name, per the ITU and ISO specification titles is High-throughput JPEG 2000.

[peterhillman]

Question about RGB channels: I see special handling of RGB in the code. I admit I don't fully understand what's happening there, but are RGB channels in all layers treated as RGB? Would an image with channels called diffuse.R, diffuse.G, diffuse.B compress as efficiently as one with plain R, G, B? What about an image part that has multiple sets of RGB channels? The channel mapping function which seems to use the channel list to derive the mapping. Is that mapping serialized to the file to allow future changes in the mapping logic, or do both the encoder and decoder use the channel list?

[palemieux]

@peterhillman the compressor makes use of a JPEG 2000 (Part 1) feature that improves coding efficiency when the first three channels are RGB (or equivalent): in that scenario, a decorrelating transform (effectively a RGB to YUV transform) is applied to the channels before coding.

Currently both the encode and decoder determine the mapping at run time. The mapping could conceivably be serialized in the file or the the channel order modified by the encoder (if possible).

The algorithm could be modified to accommodate channels named something other than "R", "G" and "B". Does a naming convention exist?

How would multiple sets of RGB channels be signaled in the file? Is that a common use case?

More complex/flexible JPEG 2000 tools are available if needed.

[peterhillman]

Channel naming is defined in a couple of places: https://openexr.com/en/latest/TechnicalIntroduction.html#channel-names https://openexr.com/en/latest/InterpretingDeepPixels.html#channel-names-and-layers You need to look for the "base name" - the part after the last dot, if there is one, or the whole name otherwise. (If the penultimate character in the channel name is a . then you can compare the last character against R,G or B)

It's common for a multipart file to contain R G B A in one part and L1.R L1.G L1.B L1.A in another. Since the compression runs per-part it may only see L1.R L1.G L1.B L1.A, so that could be treated as an RGBA group.

Having multiple RGB layers within the same part is possible, e.g. six channels forming two layers: R G B L1.R L1.G L1.B. If the codec only supports one set of RGB channels it makes sense to use R G B if any of those exist, and fall back to the (alphabetically) first set of *.R *.G *.B channels found otherwise.

There would be a lot of compression efficiency to be gained from a codec that understood that RGB channels within each layer have similarity to each other, but you would also expect strong correlation between all the red components (say) of each layer. Perhaps a future OpenEXR codec could apply a more complex decorrelation transform itself before using JPEG2000.

Including the mapping may be useful, and it may even be useful to support custom mappings at write time. #1942 discusses this issue in lossy DWA files, though it's more critical there because it affects image quality not just file size.

[palemieux]

@peterhillman Thanks for the detailed information. Will review the code in light of the naming convention.

Would supporting custom mappings at write time require a change to the API? What kind of mappings do you have in mind?

There would be a lot of compression efficiency to be gained from a codec that understood that RGB channels within each layer have similarity to each other, but you would also expect strong correlation between all the red components (say) of each layer. Perhaps a future OpenEXR codec could apply a more complex decorrelation transform itself before using JPEG2000.

JPEG 2000 supports complex component transformations. The questions is whether it is worth involving that machinery at this time. Is there a latent or existing use case for files with, say, multiple R channels within a part?

[peterhillman]

Would supporting custom mappings at write time require a change to the API? What kind of mappings do you have in mind?

Probably an API extension, yes, ideally one that's more general so it can be used across different codecs.

JPEG 2000 supports complex component transformations. The questions is whether it is worth involving that machinery at this time. Is there a latent or existing use case for files with, say, multiple R channels within a part?

Future work, I think, and probably a new compression method as far as OpenEXR is concerned.

Storing many channels within an OpenEXR is relatively common, and those files get very large. Making compression work well in that case would be really beneficial. An example of multiple R channels is a render where different shader components (specular, diffuse, emission) or different CG lights are split out to allow interactive adjustments later without rerendering. The Beachball example sequence is 3D-Stereo image and has separate RGB layers for left and right. It stores a single layer in each part. That is a common strategy as it reduces the amount of data that needs to be loaded and decompressed before accessing a subset of all channels. However, if a codec is very efficient at compressing multiple RGB layers together, it would make sense to store all channels in a single part to reduce the total file size.

[cary-ilm]

@palemieux, well that's unfortunate for the grammar sticklers, even the Library of Congress document is a mix of "High Throughput", "High-Throughput" and "High-throughput". I do see the ISO document has "High-Throughput" in the title, so I'm fine with that. I remove the original suggestions but left a few where things should be updated.

[palemieux]

@peterhillman et al. Should the reordering currently automatically performed by the compressor be removed and instead ask the application to provide the channels in RGB order if it wants decorrelation to be performed by the compressor? That would avoid burning in the channel reordering algorithm in the compressor.

[peterhillman]

The C++ API doesn't track the order you add files to the FrameBuffer or channellist attribute, it maintains them in alphabetical order. That's also the order when they are presented to the compressor. So making the codec aware of the ordering would be tricky. There's something to be said for making the API as codec agnostic as possible, avoiding codec-specific API functions or special access patterns that change behavior. There's often ambiguity about whether such settings should be presented to the user, or handled by the software internally. It would be much simpler if the advice could be "name your channels according to the spec, and the codec will do the right thing"

[palemieux]

@peterhillman Ok. Thanks for the background. I am tempted to include the channel map in the codestream... in case the algorithm/naming convention changes in the future.