[doc] GPU-based GUI API design

[AmesingFlank]

This PR is intended to allow for discussions surrounding the APIs to be implemented in the new GUI system of taichi. Please do not merge it yet.

To view the markdown file included in this PR in a more friendly format, please visit this link

This is a very early draft. Any comments will be much appreciated.

[netlify[bot]]

:heavy_check_mark: Deploy Preview for docs-taichi-graphics ready! Built without sensitive environment variables

:hammer: Explore the source changes: da24e829d60d7c6d6029b8fd5bb7a4f739fd1107

:mag: Inspect the deploy log: https://app.netlify.com/sites/docs-taichi-graphics/deploys/60ee8743d8f6880007eafa3f

:sunglasses: Browse the preview: https://deploy-preview-68--docs-taichi-graphics.netlify.app

[AmesingFlank]

@bobcao3 Thank you very much for your comments! But I think we might be aiming at slightly different things here. I believe the current task is to implement a GUI that (a) is simple to use even for not-so-experienced programmers and (b) does not suffer from the severe performance issues of the old ti.GUI. Integrating taichi with a powerful GUI system together with other awesome real-time rendering technologies is a fascinating task but I don't think that's top priority right now.

[bobcao3]

@bobcao3 Thank you very much for your comments! But I think we might be aiming at slightly different things here. I believe the current task is to implement a GUI that (a) is simple to use even for not-so-experienced programmers and (b) does not suffer from the severe performance issues of the old ti.GUI. Integrating taichi with a powerful GUI system together with other awesome real-time rendering technologies is a fascinating task but I don't think that's top priority right now.

I believe that no matter what this API needs to be extendable and possible to be performant. Once you offer a API to the user you can't limit what they can come up with, and it is very hard to create a performant immediate mode GUI / data visualization system. I think scene graph can be easy enough to use (if you don't create a hierarchy it is as simple as the existing API, just that it's not immediate)

[k-ye]

Thanks @bobcao3! Let's just stick with the current API design, which is mostly consistent with how ti.GUI works currently. Note that the goal of this GUI system is to ship a basic solution in Taichi for users to visualize 3D results. It's not meant to become a full-blown real-time renderer.

[bobcao3]

If I did not call scene.clear upon next frame, will the geometries from last frame persist? Which means we will need to keep a copy every frame

On Wed, Jul 14, 2021, 6:41 PM Dunfan Lu @.***> wrote:

@AmesingFlank commented on this pull request.

In website/docs/lang/articles/misc/new_gui.md https://github.com/taichi-dev/docs.taichi.graphics/pull/68#discussion_r670063105 :

+import taichi as ti

+window = ti.ui.Window("Amazing Window",res) +canvas = window.get_canvas() +scene = ti.ui.Scene() + + +while window.running:

• events = window.get_event()
• if ev.action == ti.ui.ACTION_PRESSED and ev.key == ti.ui.KEY_SHIFT:

•  ...
  

• canvas.clear(...)
• canvas.triangles(...)
• scene.clear()

canvas.clear(): clearing the color & depth buffer of the canvas. scene.clear(): clear the geometries in the scene.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/taichi-dev/docs.taichi.graphics/pull/68#discussion_r670063105, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACY7Q5EYOOIUG3PHKMT2BELTXY4ENANCNFSM475Y7LOQ .

[AmesingFlank]

If I did not call scene.clear upon next frame, will the geometries from last frame persist? Which means we will need to keep a copy every frame … On Wed, Jul 14, 2021, 6:41 PM Dunfan Lu @.***> wrote: @AmesingFlank commented on this pull request. ------------------------------ In website/docs/lang/articles/misc/new_gui.md <#68 (comment)> : > +import taichi as ti + +window = ti.ui.Window("Amazing Window",res) +canvas = window.get_canvas() +scene = ti.ui.Scene() + + +while window.running: + events = window.get_event() + if ev.action == ti.ui.ACTION_PRESSED and ev.key == ti.ui.KEY_SHIFT: + ... + + canvas.clear(...) + canvas.triangles(...) + + scene.clear() canvas.clear(): clearing the color & depth buffer of the canvas. scene.clear(): clear the geometries in the scene. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACY7Q5EYOOIUG3PHKMT2BELTXY4ENANCNFSM475Y7LOQ .

Yes. The geometries persist if clear() is not called.

I'm not actually sure if this is the most intuitive semantics. We do really want to make the API for drawing 3d geometries immediate, so that users can simply write scece.mesh(vertices,indices,...) in the render loop. Having a scene.clear() kind of goes against the immediate mode flavor.

However, if we were to go full immediate mode, the api would become something like

while 1:
     canvas = window.begin_canvas()
     scene = canvas.begin_scene()
     scene.mesh(...)

which does avoid scene.clear() but is even more cumbersome.

Suggestions?

[bobcao3]

Begin & End has been the choice for immediate mode API for all sorts of systems. A full immediate mode API will have some performance issues, you can reference legacy GL performance

On Wed, Jul 14, 2021, 7:41 PM Dunfan Lu @.***> wrote:

If I did not call scene.clear upon next frame, will the geometries from last frame persist? Which means we will need to keep a copy every frame … <#m_-3178495068830036824_> On Wed, Jul 14, 2021, 6:41 PM Dunfan Lu @.***> wrote: @AmesingFlank https://github.com/AmesingFlank commented on this pull request. ------------------------------ In website/docs/lang/articles/misc/new_gui.md <#68 (comment) https://github.com/taichi-dev/docs.taichi.graphics/pull/68#discussion_r670063105> : > +import taichi as ti + +window = ti.ui.Window("Amazing Window",res) +canvas = window.get_canvas() +scene = ti.ui.Scene() + + +while window.running: + events = window.get_event() + if ev.action == ti.ui.ACTION_PRESSED and ev.key == ti.ui.KEY_SHIFT: + ... + + canvas.clear(...) + canvas.triangles(...) + + scene.clear() canvas.clear(): clearing the color & depth buffer of the canvas. scene.clear(): clear the geometries in the scene. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#68 (comment) https://github.com/taichi-dev/docs.taichi.graphics/pull/68#discussion_r670063105>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACY7Q5EYOOIUG3PHKMT2BELTXY4ENANCNFSM475Y7LOQ .

Yes. The geometries persist if clear() is not called.

I'm not actually sure if this is the most intuitive semantics. We do really want to make the API for drawing 3d geometries immediate, so that users can simply write ''scece.mesh(vertices,indices,...)" in the render loop. Having a scene.clear() kind of goes against the immediate mode flavor.

However, if we were to go full immediate mode, the api would become something like

while 1:

 canvas = window.begin_canvas()

 scene = canvas.begin_scene()

 scene.mesh(...)

which does avoid scene.clear() but is even more cumbersome.

Suggestions?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/taichi-dev/docs.taichi.graphics/pull/68#issuecomment-880344838, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACY7Q5HVBE3RYJ6CTU65M5LTXZDETANCNFSM475Y7LOQ .

[bobcao3]

I would suggest instead of using a,b,c separately use a single buffer / 3xN image

On Wed, Jul 14, 2021, 8:28 PM Dunfan Lu @.***> wrote:

@AmesingFlank commented on this pull request.

In website/docs/lang/articles/misc/new_gui.md https://github.com/taichi-dev/docs.taichi.graphics/pull/68#discussion_r670106034 :

+## 2D Canvas + +### Creating a canvas + +python +canvas = window.get_canvas() + +this retrieves a Canvas object that covers the entire window. + +### Drawing on the canvas + +```python +canvas.clear(color) +canvas.triangles(a,b,c,color) +canvas.triangles_indexed(positions,indices,color)

Yes. That could work.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/taichi-dev/docs.taichi.graphics/pull/68#discussion_r670106034, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACY7Q5BNKIGYDJ6R7UQKGSTTXZIWXANCNFSM475Y7LOQ .

[archibate]

For robust APIs, does taichi-dev/taichi_three helps? I used pre-allocated vbo because taichi doesn't support dynamic field allocation to prevent copy overhead, in other words only copy when necessary, keep data around on gpu. For the need deep copy every frame problem, you could use the 'id' method, like scene.update_riangle(0, a, b, c) only updates the vbo of primtive No. 0. And if not called the triangle will be resistant in gpu memory and thus more efficient.

无法顺畅的大口呼吸，是活着的最好证明

---Original--- From: "Dunfan @.> Date: Thu, Jul 15, 2021 11:28 AM To: @.>; Cc: @.***>; Subject: Re: [taichi-dev/docs.taichi.graphics] [doc] GPU-based GUI API design (#68)

@AmesingFlank commented on this pull request.

In website/docs/lang/articles/misc/new_gui.md: > + +## 2D Canvas + +### Creating a canvas + +python +canvas = window.get_canvas() + +this retrieves a Canvas object that covers the entire window. + +### Drawing on the canvas + +```python +canvas.clear(color) +canvas.triangles(a,b,c,color) +canvas.triangles_indexed(positions,indices,color)
Yes. That could work.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[bobcao3]

In an immediate mode API we wouldn't really have a concept of VBO

On Wed, Jul 14, 2021, 9:19 PM 彭于斌 @.***> wrote:

For robust APIs, does taichi-dev/taichi_three helps? I used pre-allocated vbo because taichi doesn't support dynamic field allocation to prevent copy overhead, in other words only copy when necessary, keep data around on gpu. For the need deep copy every frame problem, you could use the 'id' method, like scene.update_riangle(0, a, b, c) only updates the vbo of primtive No. 0. And if not called the triangle will be resistant in gpu memory and thus more efficient.

无法顺畅的大口呼吸，是活着的最好证明

---Original--- From: "Dunfan @.> Date: Thu, Jul 15, 2021 11:28 AM To: @.>; Cc: @.***>; Subject: Re: [taichi-dev/docs.taichi.graphics] [doc] GPU-based GUI API design (#68)

@AmesingFlank commented on this pull request.

In website/docs/lang/articles/misc/new_gui.md: > + +## 2D Canvas + +### Creating a canvas + +python +canvas = window.get_canvas() + +this retrieves a Canvas object that covers the entire window. + +### Drawing on the canvas + +```python +canvas.clear(color) +canvas.triangles(a,b,c,color) +canvas.triangles_indexed(positions,indices,color) Yes. That could work.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/taichi-dev/docs.taichi.graphics/pull/68#issuecomment-880379281, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACY7Q5CXXJ3SDVA46BSUI63TXZOV7ANCNFSM475Y7LOQ .