GGUI 1-component grayscale field support

[Hyiker]

Describe the bug

By the document, GGUI Canvas set_image should support ti.field as parameter. However, when I actually passed a field in, it reports input image needs to be a Vector field.

To Reproduce

import taichi as ti

ti.init(arch=ti.cuda, debug=True)

pixels = ti.field(ti.f32)
# hierarchical layout
hl = ti.root.dense(ti.i, 400).dense(ti.j, 400)
# flat layout
# fl = ti.root.dense(ti.i, 400).dense(ti.j, 400)

hl.place(pixels)


@ti.kernel
def render():
    for i, j in pixels:
        pixels[i, j] = 0.0


if __name__ == '__main__':
    window = ti.ui.Window('Perlin Noise', (400, 400))
    canvas = window.get_canvas()
    while window.running:
        render()

        canvas.set_image(pixels)
        window.show()

Log/Screenshots

[Taichi] version 1.0.4, llvm 10.0.0, commit 2827db2c, win, python 3.10.5
[Taichi] Starting on arch=cuda

Traceback (most recent call last):
  File "C:\Users\carbene\Projects\misc\Taichi\course3\perlin_noise.py", line 26, in <module>
    canvas.set_image(pixels)
  File "C:\Users\carbene\AppData\Local\Programs\Python\Python310\lib\site-packages\taichi\ui\canvas.py", line 31, in set_image
    staging_img = to_u8_rgba(img)
  File "C:\Users\carbene\AppData\Local\Programs\Python\Python310\lib\site-packages\taichi\ui\staging_buffer.py", line 116, in to_u8_rgba
    raise Exception(
Exception: the input image needs to be a Vector field (matrix with 1 column)
GLFW Error 65537: The GLFW library is not initialized

Additional comments

I suppose it's a mistake in the doc or a forgotten feature to implement? If not, I'm expecting GGUI to add support for this, thanks.

[jim19930609]

Hi Hyiker, I managed to reproduce this issue locally and is still investigating it.

Meanwhile, you can try the following code, which works around this issue by using our GUI interface:

import taichi as ti

ti.init(arch=ti.cuda, debug=True)

pixels = ti.field(ti.f32)
hl = ti.root.dense(ti.i, 400).dense(ti.j, 400)

hl.place(pixels)


@ti.kernel
def render():
    for i, j in pixels:
        pixels[i, j] = 0.0 


if __name__ == '__main__':
    gui = ti.GUI('Sparse Grids', (400, 400))

    for i in range(100000):
        render()
            
        gui.set_image(pixels)
        gui.show()

[jim19930609]

Worked out a direct solution using GGUI interface. The thing is even if it's a gray-scale image, we have to fake a ti.Vector.field(1, ...) to make set_image() interface happy:

import taichi as ti

ti.init(arch=ti.cuda, debug=True)

pixels = ti.Vector.field(1, ti.f32)
# hierarchical layout
hl = ti.root.dense(ti.i, 400).dense(ti.j, 400)
# flat layout
# fl = ti.root.dense(ti.i, 400).dense(ti.j, 400)

hl.place(pixels)


@ti.kernel
def render():
    for i, j in pixels:
        pixels[i, j] = [0.0]


if __name__ == '__main__':
    window = ti.ui.Window('Perlin Noise', (400, 400))
    canvas = window.get_canvas()
    while window.running:
        render()

        canvas.set_image(pixels)
        window.show()

[PENGUINLIONG]

Should we get something like cmap in matplotlib, or is it simply too complicated? Currently set_image should be zero-padding those non-existing components.

[Hyiker]

Worked out a direct solution using GGUI interface. The thing is even if it's a gray-scale image, we have to fake a ti.Vector.field(1, ...) to make set_image() interface happy:

import taichi as ti

ti.init(arch=ti.cuda, debug=True)

pixels = ti.Vector.field(1, ti.f32)
# hierarchical layout
hl = ti.root.dense(ti.i, 400).dense(ti.j, 400)
# flat layout
# fl = ti.root.dense(ti.i, 400).dense(ti.j, 400)

hl.place(pixels)


@ti.kernel
def render():
    for i, j in pixels:
        pixels[i, j] = [0.0]


if __name__ == '__main__':
    window = ti.ui.Window('Perlin Noise', (400, 400))
    canvas = window.get_canvas()
    while window.running:
        render()

        canvas.set_image(pixels)
        window.show()

Finally, I solved it by patching to_u8_rgba function to make my original code work:

@ti.kernel
def copy_image_f32_to_u8_gs(src: ti.template(), dst: ti.template()):
    for i, j in src:
        c = src[i, j]
        c = max(0.0, min(1.0, c))
        c = c * 255
        dst[i, j][0] = dst[i, j][1] = dst[i, j][2] = ti.cast(c, u8)
        # alpha channel
        dst[i, j][3] = u8(255)

@ti.kernel
def copy_image_u8_to_u8_gs(src: ti.template(), dst: ti.template()):
    for i, j in src:
        c = src[i, j]
        c = max(0.0, min(1.0, c))
        c = c * 255
        dst[i, j][0] = dst[i, j][1] = dst[i, j][2] = ti.cast(c, u8)
        # alpha channel
        dst[i, j][3] = u8(255)
def to_u8_rgba(image):
    gray_scale = not hasattr(image, 'n')
    if not hasattr(image, 'm') or image.m != 1:
        raise Exception(
            'the input image needs to be a Vector field (matrix with 1 column)'
        )
    if len(image.shape) != 2:
        raise Exception(
            "the shape of the image must be of the form (width,height)")

    if not gray_scale and image.dtype == u8 and image.n == 4:
        # already in the desired format
        return image

    if image not in image_field_cache:
        staging_img = Vector.field(4, u8, image.shape)
        image_field_cache[image] = staging_img
    else:
        staging_img = image_field_cache[image]

    if image.dtype == u8:
        if gray_scale:
            copy_image_u8_to_u8_gs(image, staging_img)
        else:
            copy_image_u8_to_u8(image, staging_img, image.n)
    elif image.dtype == f32:
        if gray_scale:
            copy_image_f32_to_u8_gs(image, staging_img)
        else:
            copy_image_f32_to_u8(image, staging_img, image.n)
    else:
        raise Exception("dtype of input image must either be u8 or f32")
    return staging_img

And I guess the 1-dimension Vector solution may not be an acceptable one since set_image uses 0 to fill green and blue component.

[jim19930609]

Hi Hyiker, Thanks so much for figuring out the problem & solution, this is awesome!

Was also wondering if you may create a PR to merge your fix? Thanks!

[Hyiker]

Hi Hyiker, Thanks so much for figuring out the problem & solution, this is awesome!

Was also wondering if you may create a PR to merge your fix? Thanks!

Of course, I opened a pull request #5648 for this. And sadly, I didn't find a suitable solution for a numpy ndarray image.

Update: I found there is native support for numpy ndarray in taichi kernels, and I just used it for supporting numpy image sources. Also I found the code having a bug in the ti.ndrange and I fixed it after closing the previous PR. I opened another PR #5654 for both commits.