Texture compute support

[habemus-papadum]

There are some features that I think might be useful to add to simplify performing 'texture compute' (such as the stuff done in tensor.js). This primarily involves adding helpers for reading and writing data from textures; I don't think this requires much code or refactoring, but as I was thinking through the various issues it seemed tricky enough to first write things down and get some outside input.

High Level Goals

• Simplifying and assisting in the efficient/correct usage of vanilla/data oriented texture2D formats (R32I, RG16F, etc) See https://webgl2fundamentals.org/webgl/lessons/webgl-data-textures.html
• Trying to make support as broad as possible (3D textures, texture arrays, mipmaps, cube?), depth/stencil. other formats?

New Features

• For a given Texture (or the parameters accepted by the Texture constructor) suggest:
  • which TypedArray to use
  • the proper size for such an array at each miplevel desired
  • Data layout info
    • raw stride info for each texture dimension, respecting flipY (I never get this right)
    • Documentation
• Read texture data
  • Given an existing framebuffer and framebuffer attachment
  • Debug helper -- takes just a texture
    • Creates a new framebuffer, attaches texture, reads, and deletes fb when done, correctly saves and restores app state.
• Examples
  • Reaction diffusion or some other 2d PDE
  • Benchmarks
    • texture read/write
    • Flops for dot product and matrix mult

Gotchas It would be helpful if the following quirks could be addressed properly for the user automatically:

• https://github.com/KhronosGroup/WebGL/issues/2747
• app.gl.pixelStorei(PicoGL.UNPACK_ALIGNMENT, 1) for R8
• app.floatRenderTargets().linearFloatTextures();

Cleanup The following points came up as I was reading through the current code base

• current readframebuffer -- need a way to store and restore state similar to what is being done with drawframe buffer
• readpixel only supports reading from colorattachment0
• pixelStorei calls in Texture.resize need to be called for both read and write of pixels (with proper UNPACK/PACK variant)
• generateMipMap should be possible to trigger by users (for instance after rendering to texture)
• Should be able to specify miplevel when attaching color attachment to framebuffer (useful for reading) https://stackoverflow.com/questions/50979565/summing-the-values-in-a-webgl2-r32f-texture-by-generating-a-mipmap
• Currently framebuffer has width and height autoset to the last attachment's width and height -- want to better understand if this has unintended consequences in support multiple attachments... ask @tsherif?

Implementation Path

• Format helpers being discussed in #133
• TypedArray<-->Texture allocation helpers from 32I, 32UI, 32F only
• Read Pixels + Cleanup, pass 1
• Reaction Diffusion Example
• Pass 2 -- add broader support
• Benchmark example

Texture/Typed Array details

• For a given Texture (or the parameters accepted by the Texture constructor) suggest:
  • which TypedArray to use
    • R8 -> Uint8Array or normed...
    • Support Normed arrays
    • 16bit Float support (Half Float)
    • Support obscure bits packing e.g. RGB9_E5 see https://webgl2fundamentals.org/webgl/lessons/webgl-data-textures.html?
    • The main decision points for the users are
      • in the shader, do you want to do (signed/ unsigned) integer or (normed/unnormed) float arithmetic (e.g. R8, R32UI)
      • How much memory do you want to use to store your textures (e.g. R16F vs R32F)
      • Do you need to optimize time to marshal data to and from javascript vs keeping data in a format that is convenient for doing computation in javascript (e.g. HALF_FLOAT)
      • Given a bit precision level desired for computation, query gl parameters to auto suggest the proper precision directive to use in shader

[tsherif]

All seems pretty reasonable. For the second item would it just be a matter of changing App.readPixel to allow specifying the width and height?

[habemus-papadum]

@tsherif Not sure if github pings you when entries are edited, so adding this comment just in case.

[tsherif]

Thanks for the thorough treatment! This sounds very interesting. I'll read in more detail and make comments later, but one note is that a core part of how I've developed PicoGL is to go "example first", i.e. choose an algorithm I want to implement and build out the API to support it (I should probably document this somewhere). There are two reasons for this:

• Ensures APIs are built in a way that "feels good" in a real-world use case
• The examples act as integration tests for the library

So what I'd suggest is start by building out that reaction diffusion example and adding the API features required to support it.

[tsherif]

Oh and to answer your questions about FBOs. I believe all attachment to an FBO have to be the same size, but I'll double check that...

[habemus-papadum]

Example first sounds good!

[habemus-papadum]

So I spent some time investigating some details regarding textures, reading and writing them , how the format parameters work etc. Ultimately I should write a one or two page tutorial regarding the details that are important for machine learning and physics simulations, but for now I'm going to place bits and pieces here.

• in the call to texSubImage2d (for example, similar points apply to 3D, etc) there are three format parameters: InternalFormat, Format, Type in addition to the data parameter -- which for now we will always assume is a TypedArray.
• in readPixels there is only Format, Type, plus a TypedArray data parameter
• Format and Type (I believe) are only relevant to specifying how data should be marshalled into and out of the TypeArray. Once the marshalling is done, they are no longer relevant. I'm ~80% confident that's how this works.
• InternalFormat will govern how much GPU memory is used, whether shaders should use isampler2D, usampler2D, sampler2D, etc
• With regards to marshalling, the basic questions are which TypedArray to use, how many elements does it need to have, and in come cases how to encode and decode data into the proper bit format.
• The mapping from Type to TypedArray is specified in section 3.7.6 Texture Objects of the WebGL2 Spec:

TypedArray  WebGL Type 
----------  ----------
Int8Array	BYTE
Uint8Array	UNSIGNED_BYTE
Int16Array	SHORT
Uint16Array	UNSIGNED_SHORT
Uint16Array	UNSIGNED_SHORT_5_6_5
Uint16Array	UNSIGNED_SHORT_5_5_5_1
Uint16Array	UNSIGNED_SHORT_4_4_4_4
Int32Array	INT
Uint32Array	UNSIGNED_INT
Uint32Array	UNSIGNED_INT_5_9_9_9_REV
Uint32Array	UNSIGNED_INT_2_10_10_10_REV
Uint32Array	UNSIGNED_INT_10F_11F_11F_REV
Uint32Array	UNSIGNED_INT_24_8
Uint16Array	HALF_FLOAT
Float32Array	FLOAT

• The number of elements in the array and precisely which elements will be read and which are skipped are controlled by FORMAT, TYPE, the texture type (2d, 3d, etc), the texture dimensions, and the pixelStorei parameters
• Again, the pixelStorei parameters are related only to marshalling data only (again, so I believe)

So one basic question that needs to be decided is that for the purposes of picogl, should a Texture keep info related to marshalling data as part of its state (as it does now), or should they be passed each time when reading and writing (and for convenience in the constructor).

My guess is that there aren't many use cases where data is being marshalled into and out of texture in multiple formats but since picogl does a good job of keeping possibilities as open as possible I figured I would at least raise the question. Probably a reasonable compromise is to always remember the last used marshalling parameters, and use them by default, if they are not specified.