*** An implementation of

*** Lehtinen, J., Aila, T., Chen, J., Laine, S., and Durand, F. 2011, *** Temporal Light Field Reconstruction for Rendering Distribution Effects, *** ACM Transactions on Graphics 30(4) (Proc. ACM SIGGRAPH 2011), article 55.

*** http://research.nvidia.com/publication/temporal-light-field-reconstruction-rendering-distribution-effects *** http://groups.csail.mit.edu/graphics/tlfr *** http://dx.doi.org/10.1145/1964921.1964950

System requirements

• Microsoft Windows XP, Vista, or 7. Developed and tested only on Windows 7 x64.

• For filtering large sample sets, several gigabytes of memory and a 64-bit operating system.

• For GPU reconstruction:

  • NVIDIA CUDA-compatible GPU with compute capability 2.0 and at least 512 megabytes of RAM. GeForce GTX 480 is recommended.

  • NVIDIA CUDA 4.0 or later (see http://developer.nvidia.com/cuda-toolkit-archive)

  • Microsoft Visual Studio 2008. Required even if you do not plan to build the source code, as CUDA compilation, which happens at runtime, requires it.

• This software runs and compiles only on Windows and Visual Studio 2008. We welcome contributions of ports to other versions of Visual Studio and other OSs.

Instructions

General use

Launching reconstruction_app.exe will start the viewer application, which by default loads a sample buffer from the data/ directory. The default view (F1) shows the input samples with box filtering, which results in a noisy image.

Pressing F3 (or clicking the corresponding button in the GUI) will run our reconstruction algorithm and display the result.

Pressing space will toggle between CPU and GPU reconstruction. This only works provided that you have a CUDA-capable GPU with compute capability 2.0 or over.

Pressing F5 will run the reconstruction algorithm to produce a non-blurry pinhole image at the end of the time interval (t=1). This is for debugging purposes. Shading is in general not identical to the ground truth pinhole image (F4): If, for example, the scene has motion, the pinhole reconstruction is done from samples that include motion blurred shadows, whereas the ground truth pinhole image is rendered from a static setup.

If available, the app loads in a ground truth image from the same directory as the sample buffer. F2 allows you to view it. File name must be exactly as shown in the provided example. The two single digits in the filename specify a gamma the images were rendered with, so the app can apply the same to its own output (this just sets the gamma slider on the right to the specified value). The gamma is only read from the non-pinhole reference, the pinhole reference must match.

Sample Buffer Format

Sample buffers are stored in two files: the main file that contains the sample data itself, and a second header file, whose name must match the main file. For example,

samplebuffer.txt
samplebuffer.txt.header

form a valid pair.

The header specifies all kinds of useful information. It looks like this:

Version 1.3
Width 962
Height 543
Samples per pixel 16
Motion model: perspective
CoC coefficients (coc radius = C0/w+C1): -15.242497,19.053122
Encoding = text
x,y,z/w,w,u,v,t,r,g,b,a,mv_x,mv_y,mv_w,dwdx,dwdy

Width and Height specify the dimensions of the image, in pixels. Samples per pixel says how many samples to expect. Motion model is there for historical reasons; only value currently supported is "perspective". The CoC coefficients give formulas for computing the slope dx/du and dy/dv given the camera space depth (w) for a sample (see below). The last row serves as a reminder on how to interpret the numbers in the actual sample file.

The actual sample data file can be either text or binary. We recommend generating your sample buffers in text format and using the functionality in UVTSampleBuffer to convert it to binary; this can be done by loading in the text version and serializing back to disk with the binary flag turned on.

The "CoC coefficients" C0,C1 are constants that are used for computing the circle of confusion for a given depth, assuming a thin lens model. The CoC corresponds directly to the slopes dx/du and dy/dv for a given depth w. This is how to compute C0 and C1, illustrated using PBRT's perspective camera API:

float f = 1.f / ( 1 + 1.f / pCamera->getFocalDistance() );
float sensorSize = 2 * tan( 0.5f * pCamera->getFoVRadians() );
float cocCoeff1 = pCamera->getLensRadius() * f / ( pCamera->getFocalDistance() - f );
cocCoeff1 *= min( camera->film->xResolution, camera->film->yResolution ) / sensorSize;
float cocCoeff0 = -cocCoeff1 * pCamera->getFocalDistance();

If you use another model, you must derive the constants C0 and C1 yourself.

In the main sample file, each line describes one sample specified by 16 floating point numbers.

x and y are the sample's pixel coordinates, including
fractional subpixel offsets.

z/w is projected z, which is currently unused.

w is the camera-space depth.

u and v are the lens coordinates at which the sample was
taken, in the range [-1,1].

t is the time coordinate in the range [0,1] denoting the
instant the sample was taken.

r,g,b,a is the sample's radiance, in linear RGB. alpha is
currently unused.

mv_x, mv_y and mv_w are the sample's motion vector. They
encode the difference of the camera space (homogeneous)
position of the sample at the end of the shutter interval
(t=1) and the beginning of the shutter interval (t=0). In
other words, it must satisfy

(xy*w)(T=0) = xy(T=t)*w(T=t) - t*V
(xy*w)(T=1) = xy(T=t)*w(T=t) + (1-t)*V

Where xy(T=t) and w(T=t) mean the sample's original screen
position and depth at the time it was taken. Notice that the
screen position is converted to homogeneous coordinates by
multiplication by w, but there is no scale and bias to [-1,1]
clip space coordinates; dividing by w yields pixel coordinates
directly.

NOTE that pbrt's default motion model is not world-affine as
it performs interpolation of rotation matrices. As mentioned
in the paper, we changed this in our version. A pbrt patch
that outputs sample buffers with world-affine motion will be
released separately.

See Sec. 3.1 of the paper for a more thorough explanation.

CUDA

The pre-built 64-bit binary is naturally built with CUDA support, but it is by default disabled in the code to enable building without the CUDA SDK. To enable CUDA support, find the line

#define FW_USE_CUDA 0

in src/framework/base/DLLImports.hpp and change it to

#define FW_USE_CUDA 1

Provided you have a working install of CUDA Toolkit 4.0, this will enable the GPU code path (toggled by Spacebar in the application). The first time it is run, the application will compile and cache the .cu file containing the reconstruction kernels. This may take a few seconds.

If you get an error during initialization, the most probable explanation is that the application is unable to launch nvcc.exe contained in the CUDA Toolkit. In this case, you should:

• Set CUDA_BIN_PATH to point to the CUDA Toolkit "bin" directory, e.g. "set CUDA_BIN_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v4.0\bin".

• Set CUDA_INC_PATH to point to the CUDA Toolkit "include" directory, e.g. "set CUDA_INC_PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v4.0\include".

• Run vcvars32.bat to setup Visual Studio paths, e.g. "C:\Program Files\Microsoft Visual Studio 9.0\VC\bin\vcvars32.bat".

Release notes

• The soft shadow filtering code is included for reference, but not directly callable from the example application. This is because it requires a significant amount of support code for generating the light samples from camera samples, etc. We apologize for the inconvenience.

• DEVIATION FROM THE PAPER: Hieararchy consumes less memory than reported due to code cleanup.

Known issues

Fast motion with large depth differences:

Objects that undergo fast motion during the shutter interval, such that they move significantly towards or away from the camera, cause the samples' apparent screen trajectories x(t) and y(t) to deviate from straight lines in XYT. Because of this curvature, the BVH nodes' bounds become looser -- this is because we use linear XYUVT hyperplanes we use as the bounds. This in turn may lead to somewhat reduced efficiency, particularly in the CUDA implementation which has strict bounds on the number of tree nodes it can handle during the filtering. Exceeding this bound causes it to revert back to the CPU implementation. We have not observed this behavior in anything but in test cases where the motion in the Z direction is large.

Using bounds that better adapt to the perspective-induced curvature in the trajectories should fix this problem if it becomes a practical issue. One such formulation can be found in our paper "Clipless Dual-Space Bounds for Faster Stochastic Rasterization", ACM TOG 30(4) (Proc. SIGGRAPH 2011).