Passing in an Array of Textures

[ilan-gold]

Actual Result

A black screen, indicating that the data is not making it to the fragment shaders (undefined sampler?)

Expected Result

A yellow(R+G?) block (See below).

Reproduce Steps

This might just be a question of how to do something, so I apologize if this is not a bug, but I am wondering how to pass in an array of textures to a fragment shader via luma. It doesn't seem to work how I expect out of the box, but I also might have this wrong or not know about a lumagl setting/parameter (both likely). Please forgive my bad WebGL code, I am still using my proverbial training wheels.

import {AnimationLoop, Model, Geometry} from '@luma.gl/engine';
import {Buffer, Texture2D, clear} from '@luma.gl/webgl';
import GL from '@luma.gl/constants';

function makeArray(arrLength, max, i) {
    var arr = new Array(arrLength).fill(0);
    return arr.map(function(e) {
        if(i % 3 == 0){
          i += 1
          return max
        } else {
          i += 1
          return 0
      }
  });
}

var picSize = 256;
var arr = [new Uint16Array(makeArray(picSize * picSize * 3, 65535, 3)), new Uint16Array(makeArray(picSize * picSize * 3, 65535, 2))]

const vs = `#version 300 es
void main() {
  gl_PointSize = 1000.0;
  gl_Position = vec4(0, 0, 0, 1);
}
`;
const fs = `#version 300 es
precision highp float;
precision highp int;
precision highp usampler2D;


// our texture
uniform usampler2D u_image[2];
// range

out vec4 color;

void main() {
  uvec4 hold = uvec4(0.0);
  for(int i = 0; i < 2; i++){
    hold +=  texture(u_image[i], gl_PointCoord.xy);
  }
  color = vec4(vec3(hold) / float(65535), 1.0);
}
`;

const loop = new AnimationLoop({
  onInitialize({gl}) {
      const texture0 = new Texture2D(gl, {
        width: picSize,
        height: picSize,
        format: GL.RGB16UI,
        dataFormat: GL.RGB_INTEGER,
        type: GL.UNSIGNED_SHORT,
        data: arr[0],
        mipmaps: false,
        parameters: {
          [GL.TEXTURE_MIN_FILTER]: GL.NEAREST,
          [GL.TEXTURE_MAG_FILTER]: GL.NEAREST,
        }
      });
      const texture1 = new Texture2D(gl, {
        width: picSize,
        height: picSize,
        format: GL.RGB16UI,
        dataFormat: GL.RGB_INTEGER,
        type: GL.UNSIGNED_SHORT,
        data: arr[1],
        mipmaps: false,
        parameters: {
          [GL.TEXTURE_MIN_FILTER]: GL.NEAREST,
          [GL.TEXTURE_MAG_FILTER]: GL.NEAREST,
        }
      });

      var model = new Model(gl, { vs, fs,
        geometry: new Geometry({
          id: 'image',
          drawMode: GL.POINTS,
          vertexCount: 4
        }),
        uniforms: {
          u_image: [texture0, texture1]
        }
      })

    return {model};
  },

  onRender({gl, model}) {
    clear(gl, {color: [0, 0, 0, 1]});
    model.draw();
  }
});

loop.start()

Log:

uma.gl: luma.gl 8.0.1 - set luma.log.level=1 (or higher) to trace rendering init.js:48:123
luma.gl: WebGL2 context (ATI Technologies Inc.,AMD Radeon Pro 560X OpenGL Engine) context.js:210:113
luma.gl: Feature: TIMER_QUERY not supported features.js:43:129
Error: WebGL warning: validateProgram: Implemented as a no-op on Mac to work around crashes. bundle.js line 1544 > eval:381:12
Error: WebGL warning: drawArrays: Drawing without vertex attrib 0 array enabled forces the browser to do expensive emulation work when running on desktop OpenGL platforms, for example on Mac. It is preferable to always draw with vertex attrib 0 array enabled, by using bindAttribLocation to bind some always-used attribute to location 0. bundle.js line 1544 > eval:200:23
Source map error: Error: NetworkError when attempting to fetch resource.
Resource URL: webpack:///./node_modules/@luma.gl/webgl/dist/esm/init.js?
Source Map URL: init.js.map

Source map error: Error: NetworkError when attempting to fetch resource.
Resource URL: webpack:///./node_modules/@luma.gl/gltools/dist/esm/context/context.js?
Source Map URL: context.js.map

[WDS] Live Reloading enabled. client:52:9
Source map error: Error: NetworkError when attempting to fetch resource.
Resource URL: webpack:///./node_modules/@luma.gl/webgl/dist/esm/features/features.js?
Source Map URL: features.js.map

[ilan-gold]

Hello,

Is this being looked at? I would be interested in contributing this if it is a low priority. Otherwise, I'd just love to know where this stands. Let me know if you need anything else!

Thanks - I love this suite of tools so much!

Ilan

[tsherif]

@ilan-gold Apologies have been meaning to look into this, but have been busy. I can take a look next week, but you'd like to take a crack at it yourself, that would be great!

[ilan-gold]

Thank you so much, I'd appreciate you taking a look. I'll check in again and if you're still too busy next week, I hopefully will have time to look into this. Do you know of any technical limitations with luma.gl that would prevent this from working? I've gotten arrays of samplers working outside luma.gl so hopefully it shouldn't be too much :)

[ilan-gold]

Have you had a chance to look into this? If not, I can take a swing at this as best I can.

[1chandu]

One possible issue could be, RGB16UI is not fully supported (this is WebGL2 only, so assuming you are using WebGL2 supported browser), there might be some work needed in texture related classes/utility methods in luma.gl. To isolate the problem to format support, you can try RGBA/UNSIGNED_BYTE or RGBA32F/FLOAT formats to make sure rest of the code is correct.

To explore more on the support for RGB16UI, you can look at texture.spec and texture-formats table.

[ilan-gold]

Hi, this kind of fell by the wayside for me, but I might get around to it in the near future. I currently use R16UI with no issue in luma.gl as far as I can tell. The Texture2D class exposes the right parameters to let you override what you need to. Thanks for the advice - hopefully I get a chance to work on this soon!