openexr
openexr copied to clipboard
AcademySoftwareFoundation
new attribute needed to detail exact physical dimensions corresponding to display window
Various packages that reconstruct camera position and orientation from image sequences require their users to enter the position and dimensions of the rectangle on the sensor or film plane that corresponds to the display window. Alternatively the dimensional specification is in pixels and the package requires the user to enter a pixel dimension (almost always a single pixel dimension, although OpenEXR is so old that it makes provision for non-square pixels).
Several camera systems provide enough metadata in their output to determine the dimensions and offset from optical sensor of this rectangle, or provide enough that when combined with static pixel dimensions from the manufacturer allow both dimensions and offset to be computed. Depending on the sophistication of the VFX artist or facility this can be easy or very tedious.
Defining a standard optional attribute to carry this rectangle's position and dimensions would encourage camera vendors and vendors of systems transcoding camera output to OpenEXR to properly populate it.
Note that this corresponds to the ActiveSensorPhysicalDimensions metadatum defined in SMPTE's camdkit.
Apologies to OpenEXR, my comment about it being "so old that it makes provision for non-square pixels" was out of line. The provision is for anamorphic acquisition and isn't some pre-digital-television non-square pixel thing.
Nick's comment in the TSC meeting yesterday about "just divorce yourself from the existing coordinate systems" [paraphrased] is correct, or at least mostly so. I think we want to retain the data window / display window convention of how you move when you increase either x or y when we talk about centering offset and about what part of the sensor is being used.
Proposal:
sensorPhotositePitch, µm, as a float
sensorOverallDimensions, mm, v2f
sensorCenterOffset, mm, v2f
sensorAcquisitionRectangle, photosites, Box2i
Will start work on a PR.
Mr Edge Case here again: would the sensorCenterOffset be relative to the optical center of the lens or is just the relationship between the camera body lens mount and which part of the sensor contributes to the final image? So if a shift lens is used (or just a badly aligned one), the sensorCenterOffset should change? Or is there already sufficient additional metadata that describes the lens properties relative to the lens mount? Most digital cameras offer a choice of aspect ratios and resolutions. I believe some cameras do resolution changes on the sensor itself using something like hardware pixel binning which might make the term 'photosite' ambiguous. Should those cameras adjust the sensorAcquisitionRectangle and sensorPhotositePitch accordingly if they aren't capturing the full sensor at full resolution? Also, what happens to all of these if there's a post-process "optical" effect, such as an aspect ratio change, or a pan or zoom? Should these parameters refer to the values "at capture time" or is image processing software expected to update these accordingly? With the philosophy that wrong metadata is worse than no metadata, I wonder if those attributes could be tagged as "not safe to preserve if the image geometry is modified" via some new API, so software that reads/modifies/writes OpenEXR images knows to drop them.
(I think all of these parameters are a good idea, and wouldn't suggest adding any more to handle the above cases, but I think a certain amount of pedantry may be required in defining them to ensure they are interpreted consistently)
Echoing Peter's comment, Is the term "photosite" right? I think of hardware sensors when I think of photosite, but maybe I'm thinking about that wrong. Is photopitch vs a Bayer green, red or blue? Or does "photosite" mean the resampled/regularized data we've got in the EXR file? Pedantically is it correct in both cases? I guess I'm unclear if photosite refers to final sampled output, or if it refers to a hardware characteristic. Since the term is being used with respect to the sensor I'm unable to work out what's right.
My understanding is that sensorCenterOffset is relative to the lens optical center, otherwise how are we to understand the direction light was coming from when it hit the sensor?
With regards sensorAcquisitionRectangle, I assume the box itself is in the coordinate system defined by the with an origin of sensorCenterOffset and the euclidean metric is defined by sensorPhotositePitch. With that information, and an unambiguous definition for sensorPhotositePitch, for any pixel I can work out any interesting fact I can think of that doesn't involve further knowledge about the lens.
I will try and take these comments in order. But before I do that I should say that I am copying concepts from SMPTE RDD 55, "Material Exchange Format — Carriage of ARRI Camera System Metadata", because I worked on that document with some very smart people and mostly I think we got things right. Mostly (see below).
sensorCenterOffset is relative to the optical center of the camera's lens mount. I have asked (and have a couple more people to ask) but have yet to find anyone tracking an offset between the optical axis of a lens relative to the center of the lens mount on the lens side, nor any sort of provision for a too-heavy lens mounted on a camera with a weak lens mount and too-weak support rods.
Regarding tilt lenses, I would not say they are never used, but as far as I know, they are almost never used for motion picture or television work. Moreover, looking at the lenses on the Wikipedia page to which you linked, the weirdness is solely on the lens side. I don't believe any of those cameras had their film gate or their sensor displaced in any way. Thus I think that if the industry cried out for metadata supporting tilt or shift lenses, we would support them by adding additional metadata beyond what is proposed here, with no need to change what this proposal provides.
I do need to come up with an exact definition relating signs of the offset components to the contents of the data window, e.g. positive x and y offsets would shift the contents of the data window up and to the left.
sensorAcquisitionRectangle refers, roughly speaking, to those photosites that contribute to the image, even if the image that is recorded does not comprise an image where the number of image rows and image columns match the number of photosite rows and photosite array columns. For example, when shooting for HD delivery on the original ALEXA EV, a 2880 x 1620 photosite array contributed sensel data [the read-out-contents of photosites] to a debayer algorithm that simultaneously downsized to a recorded 1920 x 1080 array of RGB pixel data.
In light of Nick's comments it's worth my explicitly adding a comment block saying that photosites are the rectangular (typically square) sites where the photon-induced charge is registered; sensels are the data read out of those sites; and pixels are the data that are stored in OpenEXR files. In the RGB case, the pixels result from feeding sensel data and a description of the color filter array into a demosaicing algorithm and possibly linearly scaling and/or converting an integer image to float; in the monochrome case, there is only the possible linear scaling and/or conversion from image to float.
Here is the wording from SMPTE RDD 55: "...the rectangular region of the sensor containing the photosites from which the captured image is read".
Guess what? That's incorrect. It's incorrect because most debayering algorithms work by examining adjacent pixels on either side of the pixel being reconstructed. When the debayer is working with a 5x5 array of sensel data, in the example above, technically, the examined region is 2884 x 1624.
Therefore in the next revision of RDD 55 we are going to change the language to be something like this: "...representing the rectangular region of the sensor containing photosites the contents of which are in one-to-one correspondence with the captured sensels, for a monochrome sensor, or with the reconstructed pixels, for an RGB sensor." Thought I might change "RGB sensor" to "sensor covered with an RGB color filter array arranged in a Bayer pattern".
(The Sony F65 has the duck-billed platypus of CFA-covered sensors in that it is tilted at a 45º angle)
What RDD 55 calls SensorPixelPitch I propose to call sensorPhotositePitch. The definition is "Distances between centers of sensor photosites, in microns".
What RDD 55 calls OverallSensorDimensions I propose to call sensorOverallDimensions. The description can be almost copied one-for-one; I would go with "Dimensions of the light-sensitive area of the sensor, in mm, independent of the subset of that region from which image data are obtained."
If I maintain the order of attributes in the file to be as it was in the original message here, then I would add just above the center offset wording to the effect that for the purposes of center offset, the origin of the coordinate system in which the offsets are given is, as with the data and display windows, in the upper left; and that the directions in which the row and column indices of that same coordinate system increase are the same directions as those found in the data and display windows.
In re: pedantry, apologies if I've used this setup before; It really should be on my gravestone:
DEVIL: ...and this is the lake of lava you will be spending eternity in
ME: Actually, since we're underground, it would be magma
DEVIL: You do understand this is why you're here, right?
Submitted as PR #1489 .
n.b. running locally, the added standard attribute tests passed, but two compression tests failed when I ran make test on my late-2021 MacBook Pro (i.e. Apple Silicon, if that matters) — OpenEXRCore.testDWAACompression and OpenEXRCore.testDWABCompression.
Thanks for all the explanatory notes, very much appreciated.
The test failures are unrelated to adding attributes ~ Kimball has a separate fix up for that.