KhronosGroup
KHR_image_formation
The purpose of this extension is to fully define how output pixels should be colored, as the current glTF spec only describes how to calculate the light output for each pixel in physical units. This extension provides a means to specify the transfer function to the limited, unitless range of an sRGB output format, as well as specifying default behavior that matches what most renderers are already using.
The techniques employed are leaning on popular existing standards: camera-style exposure, default ACES tone mapping and custom Adobe LUTs. In this way, the render can be set up to approximate a photographer's workflow as closely as possible.
This extension is an alternative to #2083, where the clamped, linear output described therein can be achieved by specifying an identity LUT. This extension does not provide special handling for newer HDR output formats, however it could be easily extended to provide different LUTs for different output ranges, as the film industry does today.
Hi @elalish and nice to see your entry in this area :-)
This extension is an alternative to #2083, where the clamped, linear output described therein can be achieved by specifying an identity LUT. This extension does not provide special handling for newer HDR output formats, however it could be easily extended to provide different LUTs for different output ranges, as the film industry does today.
I would say that this extension (khr_image_formation) is not an alternative to khr_displaymapping_pq as they have quite different approaches. It does not seem to me that they are trying to solve the same usecase - from what I can tell this extension has a focus on a photographers workflow, with a heavy influence of color grading and artistic intent by means of the LUTs.
This is very different from khr_displaymapping_pq and "the 3D Commerce usecase" where the whole purpose is to provide a 'neutral' mapping of content to the output.
If color grading (exposure factor and LUT) are to be applied, that is done as a final step when content is authored to the target. The same content will have different grading (exposure factor and LUT) depending on if the destination is a movie theater, DVD, broadcast, computer game or Web-viewer. Introducing grading into the asset will lead to conficting intent, for instance when mixing assets and/or when viewing on a destination that has different grading.
In my opinion color grading (exposure factor and LUT) does not belong in a glTF asset, if needed it should be in the device that consumes the content (ie knows about the output characteristics)
(For more technical details see my comments)
A LUT can be used to apply "neutral" mapping of content to the output. I'm not exactly fond of trying to talk about neutral mapping because image formation extends beyond stimulus (esp. given the limitations of displays and the HVS). One benefit of using LUTs is that it allows authoring of "neutral" scenes (with no tone mapping at all) and of artistically crafted scenes as well (high contrast black and white for instance).
Exposure in this case should not depend on the output device, but on the scene itself.
I don't see the mixed-asset issue as being a problem. It assumes that mixing assets is going to be the final step in publishing and I can't see a situation where that would happen. I see this extension as something that a viewer or exporter would support but not something that an importer would probably read.
For example, I work on Adobe Substance 3D Stager (a staging app used for synthetic photography among other things). We support importing and exporting glTF but our users set up their lighting within the application so there's not much need to support importing this info. However, when exporting or publishing directly to the web, including everything needed to render the scene as expected is very important. This includes tonemapping in addition to lighting. They go hand in hand.
Adding my notes from the call here again and some more thoughts around this, as we've been running into many issues regarding viewers and their default display of models in general:
- I agree with @rsahlin that this brings viewer properties into the asset level, which is, for many usecases, undesired, but for others it's great.
- I think that the KHR_outputmapping_pq extension has a lot of the same problems, and also might "break" viewers even more if multiple assets have different pq values, but maybe I'm not understanding it right yet.
- while I can see that ACES has been adopted in many places it isn't exactly a great/neutral tonemapping (it's hard to get some brand colors to look right with ACES)
- any kind of maximum value (e.g. the current number of 10.000 lux) is unclear to me - e.g. sunlight can be 100.000 lux, how would I represent a scene with a sun and a "daylight lamp"?
- I think artistic intent should not be expressed in a glTF file at all, and for that reason also don't think LUTs should be part of glTF assets, unless as "rendering hints that can safely be ignored". From experience with artists I can say that when you start adding the ability to use LUTs embedded in assets the assets start "breaking" - the LUT might be used instead of actually specifying proper color/light/texture values, for example.
.
- I agree with both of you that an extension is needed to provide better "rendering hints" for viewers displaying assets.
- I actually think these hints should be per camera, not per file, because different cameras might have totally different requirements for how to show a scene (e.g. a camera that is close to a light source will need to have different exposure than a camera that looks from afar)
- there are other hints to be considered, for example if the viewer should use their own / a user-provided / developer's randomly chosen IBL and tonemapping, or use "none", or use one from a specific file (I think there are some other extensions dealing with this)
To summarize:
- I'd rename the extension as something like KHR_rendering_hints to make it very explicit what the idea here is, and also make it more clear in the description that this is by no means something every viewer should/must use (e.g. AR will often be rendered differently and shouldn't care for any rendering_hints)
- some properties make sense per-camera, not sure if that should be a separate extension then.
- other properties (such as if the viewer should add its own default lights or not, or if the viewer should add its own IBL or not) might also be desired.
- while I can see that ACES has been adopted in many places it isn't exactly a great/neutral tonemapping (it's hard to get some brand colors to look right with ACES)
ACES has many issues. Two of the most glaring ones to me are the hue skews created when going back from AP1/AP0 back to sRGB, and the overall drive to yellow-ish.
- any kind of maximum value (e.g. the current number of 10.000 lux) is unclear to me - e.g. sunlight can be 100.000 lux, how would I represent a scene with a sun and a "daylight lamp"?
Right, that's why some kind of tone mapping (which I prefer to refer to as range compression, because that's really what we're trying to do) is needed and can be baked into a LUT. The reason why we chose to use a LUT was to allow authors to pick their compression curve (for instance to match an existing ACES based workflow).
- I think artistic intent should not be expressed in a glTF file at all, and for that reason also don't think LUTs should be part of glTF assets, unless as "rendering hints that can safely be ignored". From experience with artists I can say that when you start adding the ability to use LUTs embedded in assets the assets start "breaking" - the LUT might be used instead of actually specifying proper color/light/texture values, for example.
The original proposal @elalish and I craft was not using a LUT but defined a specific (yet configurable) range compression (tone mapping) curve with an extra step to somewhat control hue skews. Should we share this proposal? Note: that configurable curve can be configured to match ACES's compression.
- I actually think these hints should be per camera, not per file, because different cameras might have totally different requirements for how to show a scene (e.g. a camera that is close to a light source will need to have different exposure than a camera that looks from afar)
That's a good point.
Some great discussions here. I'd issue one incredible and loud warning...
There's a tremendous conflation here between notions of Stimulus versus Perceptual Appearance. The CIE lists two definitions for "colour" for good reason. One describes the psychophysical stimulus specification of colour, and one describes the perceptual appearance of colour.
@elalish has wisely cited that no representation in an image formed can even remotely produce the range of stimulus present "as though we were looking at it".^1 And even if this were the case, this is a seductive misnomer as cited by MacAdam, Jones, Judd, and many other researchers; an image is not ground truthed against an idealized reproduction of stimulus.
As a general cautionary rule of thumb, it should be noted that psychophysical stimulus will always be nonuniform with respect to perceptual appearance.
That is, chasing the dragon of 1:1 stimulus will always assert that the appearance of the thing in question will be incorrect. Not that it matters, because it is fundamentally impossible to replicate the stimulus from the mediums in question, and the subject of distortions of "hue" and other facets will take front and centre stage without properly considering the act of image formation as a whole.
- This covers even a theoretical optimal HDR display. The role of an image is not mere simulacrum of human perception.
@sobotka I'm curious, besides the warning, would you have any suggestions on how to resolve this in the context of glTF (and related cases such as e-commerce)? E.g. "viewers should just all auto-expose and try their best to somehow present in a somewhat considered neutral way"?
(I value your input a lot, and try to read all your Twitter threads, even when I sometimes need a number of dictionaries to understand them)
@hybridherbst Exposure is the easiest and least interesting part of the problem. The complex steps are about generating the desired perception, and to Troy's point, focusing on the stimulus is not enough. Based on many long conversations with Troy, that's why I was coming back to a LUT based approach because it will allow for future improvements in image formation as there is no great known solutions at the moment (although LUTs have the issue of being spatially invariant, which flies a bit in the face of proper perception generation if we'd want to take things like lightness into consideration; also see Bart Wronski's recent article on localized tone mapping).
The original version of the proposal I mentioned earlier was attempting to create a somewhat neutral "tone mapping" based on efforts from Troy and others, while taking into accounts constraints like applying the result in real-time without using a LUT for certain glTF viewers. Here are a couple of examples. In each image, from top to bottom: linear (no compression), ACES (SDR/Rec.709 100nits target, using the reference implementation), the proposal.
The proposal is far from perfect. There's no good gamut mapping step so it leaves a few hue skews behind but it's a noticeable improvement over ACES (blue shifting to purple is greatly improved for instance).
would you have any suggestions on how to resolve this in the context of glTF (and related cases such as e-commerce)? E.g. "viewers should just all auto-expose and try their best to somehow present in a somewhat considered neutral way"?
I tend to think that when we are talking about selling red plastic chairs, we are really firmly anchored in the realm of a perceptual appearance approach; we want to walk into a big box shop and see the brightly orange chair as the brightly orange chair and spend our fifty bucks on it. But let's ignore the appearance side, and focus purely on ratios of tristimulus...
The points that @elalish and @romainguy are raising are hugely significant here; some semblance of an open domain tristimulus compression is required.
Suggesting a virtual intermediate encoding capped at 10k nits is going to be in direct opposition to the goal of selling an orange plastic chair. We can't just "scale down" via an exposure, as that pressures the middle grey value down, which means we have to bring it back up again or our orange plastic chair is "too dark". Worse, if we view the albedos of the chair in direct sunlight, now we have a clip skew happening at the 10k nit mark without some tristimulus compression.
There's no real in-between here. and if we ignore the whole gargantuan rabbit hole of appearance modelling in image formation, we simply cannot have our "middle-ish value" of the range in the "middle-ish value" of the image and compress down with generalized "exposure".
The orange plastic fifty dollar chair should appear orange and plastic so that someone can buy it for fifty dollars. That's impossible without some care and attention to the image formation chain such as what @romainguy and @elalish are bringing to the table here.
Regarding comment from @MiiBond
I see this extension as something that a viewer or exporter would support but not something that an importer would probably read.
To me, this is a strong argument against having this extension (exposure + LUT) inside the glTF asset. I believe it shall be applied as a last step in engines,viewers or renderers, just before 'consumption'.
Just like content is authored to different targets - movie, print or web.
This is very different from khr_displaymapping_pq and "the 3D Commerce usecase" where the whole purpose is to provide a 'neutral' mapping of content to the output.
If color grading (exposure factor and LUT) are to be applied, that is done as a final step when content is authored to the target. The same content will have different grading (exposure factor and LUT) depending on if the destination is a movie theater, DVD, broadcast, computer game or Web-viewer. Introducing grading into the asset will lead to conficting intent, for instance when mixing assets and/or when viewing on a destination that has different grading.
In my opinion color grading (exposure factor and LUT) does not belong in a glTF asset, if needed it should be in the device that consumes the content (ie knows about the output characteristics)
@rsahlin Agreed we ideally need different LUTs for different destinations. Happy to include that here or in a follow-on extension. As for not belonging in the asset, I agreed in the case of an IBL extension for the same reason, but especially because of the huge over-the-wire cost that many consumers wouldn't want. This is at least compact, but yes. However, I would argue khr_displaymapping_pq has exactly the same problem: it too specifies a LUT, just a very simple one. Again, anyone who wants to post-process differently (including a metaverse/AR app) will have to ignore that extension.
Agreed we ideally need different LUTs for different destinations.
In my opinion they belong to the destination (engine) and not inside the asset. See them as a type of filter that is applied (by the application) just before consumption. Similar to how the Instagram app adds filters. Just like those filters are 'in' the Instagram app - the LUTs in this case shall be in the engine/renderer.
As for not belonging in the asset, I agreed in the case of an IBL extension for the same reason
I don't think a LUT is nowhere near like an environment light. For the following reasons:
- A LUT may be non physical - changing colors just like a filter would.
- A LUT will be fixed to one light setup. When the light changes the scene is likely to 'break'
- A LUT is not part of what I call a 3D scene. I cannot see a correlation to what we are defining with the glTF dataformat. Namely geometry, surface properties and lightsources.
- A LUT may make viewing of models from different scenes impossible while retaining the creators intension (via the LUT)
- A LUT may introduce errors in the 3D content pipeline. If the color of a model appears to be wrong, then there is nothing stopping somebody from changing the LUT to get the desired color. The apropriate behavior would be to change surface properties (basecolor). This will happen when glTFs are handle in large amounts over time.
I argue that environment light does not have any of the above problems.
I strongly believe that the way to provide a studio type of 'look & feel' on a glTF is by using the light setup (punctual and environment lights) - not by adding a filter.
I strongly believe that the way to provide a studio type of 'look & feel' on a glTF is by using the light setup (punctual and environment lights) - not by adding a filter.
You, again, are completely ignoring the role the image formation process plays in this.
There can be no moving forward until you come around to appreciating that one cannot, even if desired, output the “stimulus”.
This whole idea is nonsense.
As an idea to find a common ground here:
- I think the main point that @rsahlin wants to reach is "reproducible and consistent output"
- I think the main point that @elalish and @romainguy want to reach is avoiding an implicit incorrect LUT and implicit exposure, and instead specify exact values in the asset.
I propose:
- instead of only specifying an exposure value, introducing an exposure hint that can have one of three modes:
- "automatic"
- "fixed"
- "maxBrightness" (needs a better name - calculates exposure from scene lights similar to what KHR_displaymapping_pq is aiming to do)
- dropping the user-specified LUT either entirely or replacing that with a mode:
- "none" (current behaviour)
- "ACES" (or another name - "neutral"?, not sure @romainguy how your proposal of a neutral LUT would be named)
This way, we avoid the LUT discussion (and don't introduce too much power into the asset), and the following combinations would be possible:
Current behaviour:
"tonemapping": "none",
"exposureBehaviour": "fixed",
"exposure": "0"
Automatic exposure like many game engines do:
"tonemapping": "ACES",
"exposureBehaviour": "automatic",
"exposure": "0"
Authored fixed exposure for a daylight scene:
"tonemapping": "ACES",
"exposureBehaviour": "fixed",
"exposure": "15.0"
Similar to what KHR_displaymapping_pq would do:
"tonemapping": "ACES",
"exposureBehaviour": "maxBrightness",
"exposure": "0"
When "exposureBehaviour" is either "automatic" or "maxBrightness", "exposure" serves as an offset, allowing artistic control to nudge into a "high-key" or "low-key" look and feel.
These would all be per-camera and/or per-scene "renderer hints". Viewers could aim to match the closest camera's rendering hints if free navigation is allowed. When multiple assets are combined that all have maxBrightness mode, the behaviour is still deterministic and very clear.
(I'm not sure how far along the IBL extension proposals are; optionally this extension here could also have something like "externalLight": true / false to specify if a viewer should apply any kind of IBL or rely on lighting and/or IBL as specified in the file instead - without that, no consistent output between viewers is possible.)
I believe it would be a mistake to reduce the nonlinear portion of the mapping function to a choice of 1-2 presets, while still positioning this as a general-purpose image formation extension. If this were a proposal to define lighting and image formation scenarios specifically relevant to retail products displayed in typical stores (say, 3DC_lights_retail and 3DC_image_formation_retail?), then perhaps presets would be fine.
But for an extension meant for use in the wider glTF ecosystem, the intent of KHR_displaymapping_pq is too prescriptive, setting aside the technical objections. I do not think we should pull KHR_image_formation in that direction.
Please don't view the possibility that an artist might create a stylized LUT (rather than a "neutral" one) as a problem to be solved — it is a healthy side effect of providing the proper tools for image formation. If general consensus is that image formation definitions do not belong in a glTF file, I'm OK with that decision. But imposing more prescriptive choices on the extension will not help.
I had proposed earlier that a better name might be "KHR_rendering_hints" or something similar that makes it very clear that these are hints for viewers on how something should be displayed.
I don't have a strong opinion on including or not including user LUTs; the above was an attempt to find a useful middle ground for an extension that allows for both flexibility and the goal of getting closer to determinism.
One could also argue that one extension (e.g. KHR_rendering_hints) would be about the above (exposure, perceived brightness, a good neutral mapping for HDR values) and another extension (e.g. KHR_tonemapping) would be able to override the "neutral" mode and extend it with user LUTs. I'm not sure what the current approach is to either put multiple features into one extension or to split things up (the transmission/volume/IOR triplet seems to suggest the latter).
On optional hints —
Thanks @hybridherbst, and yes! I'm comfortable with the idea that the extension should generally be optional (formally: in "extensionsUsed" but not "extensionsRequired"). It would — eventually — be worthwhile to give examples of when we'd encourage or require a client to ignore an optional KHR_image_formation extension. Similar to how a pathtracer might discard thicknessMap rather than overriding ground truth. I probably wouldn't go as far as renaming the extension to clarify that: optional/required is a shared concern in many extensions, and KHR_materials_volume_hints is pretty verbose.
On scope —
More generally I am a bit concerned about the arc of the KHR_image_formation / KHR_displaymapping_pq discussion. We are arguing over how many legs a tripod should have. KHR_displaymapping_pq offers a tripod with one leg1, not providing the mechanism to achieve its own goal outside narrow contexts. KHR_image_formation offers a tripod with three legs, and does the job with a practical level of simplicity and flexibility. If we're going to build a tripod, we can alter the three legs (parameterize as .cube LUTs, fixed properties, .mtlx nodes, etc...), but let's not compromise on fewer than three legs. 🙏🏻
1Apologies, I know my metaphor seems to imply that this is simple and obvious — these are very complicated topics, and nothing here is obvious. But obvious or not, we need an extension that solves the problem at hand.
I agree with the above, especially the point about examples - but I also do understand Rickards concerns that this proposal currently doesn't solve the challenge he's facing - deterministic output of light values based on scene content or the combination of scenes.
Introducing "exposureBehaviour" and "exposure" as outlined would allow for these cases in my mind (files that just scale exposure based on lights in the scene and have a kind-of-deterministic view-indepdent result). What do you think about these?
When we were doing the PBR materials, we looked at the major game engines and evaluated what is the common denominator. When we were doing the PBR lights,, we looked at the major game engines and evaluated what is the common denominator.
Goal was the maximum compatibility and acceptance for glTF and the extension. Furthermore, we relied on the research and scientific papers of these companies.
I suggest, we do the same for the "last pixel mile" - exposure and tonemapping.
So, if you look at the Unreal Engine and Unity:
Unreal Engine https://docs.unrealengine.com/5.0/en-US/auto-exposure-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/color-grading-and-the-filmic-tonemapper-in-unreal-engine/ Unity https://docs.unity3d.com/Packages/[email protected]/manual/Override-Exposure.html https://docs.unity3d.com/Packages/[email protected]/manual/post-processing-tonemapping.html
Both can do manual exposure and do have the ACES tonemapping. Unity has the option for no (clipped) tonemapping. In the Unreal Enigne, it is possible to disable tonemapping as well: https://soldirix.wordpress.com/2020/05/26/ue4-how-to-disable-the-tone-mapper-once-and-for-all/
So, I suggest for this extension, to only have exposure and two well defined tonemappings:
- None
- ACES
For the LUT, I recommend to postpone it and move it to a follow up extension.
Last, but not least, having exposure = 0.0001 and tonemapping = None is the KHR_displaymapping_pq use case.
Finally, the transfer function relies on the used output colors space and render buffer and is already well defined: https://www.khronos.org/registry/DataFormat/specs/1.3/dataformat.1.3.html#TRANSFER_CONVERSION
Except this loops right back to both cases being garbage options.
When we were doing PBR next, we splitted the "beast" into several small extensions. Furthermore, we dropped and/or created new extensions in favour of compatibility and acceptance.
Compatibility and acceptance that defy the whole point of doing something in the first place is the hobgoblin of little minds.
Compatibility and acceptance that defy the whole point of doing something in the first place is the hobgoblin of little minds.
If we start with a small "lego" brick (exposure; two tonemappings), which does not block the big picture, I prefer the small solution, as it brings consensus and acceptance by the engine and tool developers. Then we do the next lego bricks.
Are you open to create an extension, which brings in the important features you mentioned before? If we do have a "northstar", we do have something where we can head to.
I think providing exposure = 0.0001 and tonemapping = None among the options (per the goals of KHR_displaymapping_pq) is reasonable, despite drawbacks of those choices for PBR rendering.
It is probably also fair to say that the classic tone mapping presets (ACES, Reinhard, etc.) are to tonemapping as a simple Phong shader is to material definitions — popular, but not particularly good. What this KHR_image_formation proposal already provides is a pretty balanced compromise between the vast flexibility of the OpenColorIO configurations you'd find in many modern tools (Blender, Maya, Unreal Engine, Houdini, Substance Painter, Arnold...) and the need to bake transforms down considerably for runtime.
I do wonder if we could go further in terms of baking — e.g. reducing the LUTs to 1D and 3D KTX2 textures with appropriate internal formats and zstd compression? But reducing this to just {ACES | None} goes too far in my opinion.
Compatibility and acceptance that defy the whole point of doing something in the first place is the hobgoblin of little minds.
@sobotka I really appreciate the detail and examples you've given above. You're right that this is a very complicated space. I doubt it's feasible to "solve" the perception problem, but the idea is to at least create consistency in output where feasible. Currently this last mile is simply unspecified; the question is can we do better than that and then build on it? Do you have any particular recommendations?
I think providing
exposure = 0.0001andtonemapping = Noneamong the options (per the goals ofKHR_displaymapping_pq) is reasonable, despite drawbacks of those choices for PBR rendering.
These values are the use case of the mentioned extension. Of course these values need to be adapted for very dark or very bright scenes.
I do wonder if we could go further in terms of baking — e.g. reducing the LUTs to 1D and 3D KTX2 textures with appropriate internal formats and zstd compression? But reducing this to just {ACES | None} goes too far in my opinion.
There are use cases e.g. embedded devices where another texture unit is a pain. If you can convince Epic Games to implement these LUT's, then I am 100% with you. Otherwise we have to wait and one of the major engines out there is not compatible if we force LUT's today.
Currently this last mile is simply unspecified; the question is can we do better than that and then build on it? Do you have any particular recommendations?
I think what is being tackled by this exact attempt is wise and prudent; think about what is seen by the audience.
In this case, and specifically in the case of E-commerce, which I believe is a sizeable portion of the problem surface, the quibbles over what is being discussed are actually sizable. Should the reddish chair look be distorted to pure yellow? Should the blue toy bunny be represented as a totally distorted purple? These sorts of things likely matter significantly in the context.
What I am cautioning is precisely what @donmccurdy more or less stated:
But reducing this to just {ACES | None} goes too far in my opinion.
And better than broken rendering for games, less E-Commerce, will only benefit.
@UX3D-nopper is there a canonical definition of ACES Filmic tone mapping somewhere? I assume we are just talking about a tone map, and not other components of ACES.
Most people probably copy & paste from here: https://github.com/selfshadow/ltc_code/blob/master/webgl/shaders/ltc/ltc_blit.fs or here: https://github.com/TheRealMJP/BakingLab/blob/master/BakingLab/ACES.hlsl
To formalize, one has to dig into the original repository: https://github.com/ampas/aces-dev They use the Color Transformation Language (CTL) https://github.com/ampas/CTL/
Basically, we need to define, how to transfer from SRGB (or another color space see https://nick-shaw.github.io/cinematiccolor/common-rgb-color-spaces.html#x34-1330004.1) to the AP1 color space, as the ACES tone mapping happens in AP1 (the RTT and ODT part). Then "go back" to SRGB (or the color space of the output buffer). One important thingy is, that AP1 has a different illuminant than SRGB, so a conversion has to happen in many cases as well:
From my perspective, converting between color spaces is already defined at many places e.g. at Khronos: https://www.khronos.org/registry/DataFormat/specs/1.3/dataformat.1.3.html#_bt_709_bt_2020_primary_conversion_example
The only missing part is RTT and ODT: https://github.com/ampas/aces-dev/blob/dev/transforms/ctl/lib/ACESlib.RRT_Common.ctl#L23 https://github.com/ampas/aces-dev/blob/dev/transforms/ctl/lib/ACESlib.ODT_Common.ctl#L32
On a first sight, it look likes one has to use the whole ACES dev package. But this is not the case. The question is, how much we need to explain RTT and ODT as it is basically "just" another tonemapping curve.
The definition you linked to are approximations and not the actual definition. The BakingLab one is a good approximation that's good for real time renderers. I've extracted a less-approximated* definition in Filament though: https://github.com/google/filament/blob/ebd5f150c16548b83bbb4a4fec9e4430c2fa1309/filament/src/ToneMapper.cpp#L28
- Less approximated because the RRT+ODT use a curve fit, just like Unity HDRP does I think. But it preserves the rest of the tweaks defined by ACES