google
RenderTarget with imported texture results in flickering
Describe the bug
Amazing to see how much work has been done the last couple of years, really awesome! After working on some other projects I'm again looking into using Filament for rendering by using a shared context. In the past I've used my own fork with some features that worked for me. Now, I want to use the import() feature of a Texture and use Filament to render a 3D scene into a texture.
In my GLFW based app I create a window and OpenGL context. I create a Filament instance and share my OpenGL context. Next I create a texture using my GL context and use the import() call. I setup a View and render it into a RenderTarget similarly as shown in the rendertarget example.
However I'm seeing flickering when rendering. I'm pretty sure that this is caused because we're having two contexts and the Filament threads/context maybe clears the shared image while I'm rendering it. Therefore I'm calling Engine::flushAndWait() which fixes the issue.
I've got some concerns regarding peformance when using flushAndWait() and the glfwSwapBuffers() so I measured the time my render loop takes when it calls flushAndWait() and I noticed no performance impact. The total time of my render loop is about 16ms but as I'm just testing and not yet rendering anything else besides the FlightHelmet scene I'm not sure if I'll run into a performance issue in the future that I can't solve due to the approach I'm using.
My question is, when I'm using an imported texture as render target of a view what is the correct way to prevent flickering?
To Reproduce
Currently I've no code example that I can share to reproduce this. I'm not sure this is a bug or if I'm using the API incorrectly. If required I'll create a standalone app that demonstrates the issue.
Expected behavior
When I render the texture that I created with the OpenGL context I would expect it not to flicker when I render it.
Screenshots
https://github.com/google/filament/assets/358809/23c3d27b-d63a-4c92-a580-c1e97565bb5f
Desktop:
- OS:
Linux 6.4.6-arch1-1 #1 SMP PREEMPT_DYNAMIC Mon, 24 Jul 2023 20:19:38 +0000 x86_64 GNU/Linux - GPU:
NVIDIA GTX 2080i - Backend:
OpenGL
Additional context
Some related issues:
- https://github.com/google/filament/issues/2261
- https://github.com/google/filament/issues/1921
Ah... yes, it's true we didn't intend for the imported texture to be used in this way. I believe the correct way to make this work is to use fences and server waits. Unfortunately we don't provide that functionality in the API.
You next best bet (which is not ideal and not 100% correct) is to use a queue of textures in stead of using just one. Basically, you would have for example 3 or 4 textures which you would use a round robin fashion to render into, while you would be reading (in your context) from an older one. This would add latency too.
The short and real answer is that this is not supported..
FYI: ARCore does something similar to what I describe here, except the other way around (they draw (from camera) and we read from).
Thanks for the info @pixelflinger. I was thinking about fences and read several mentions that this is most likely not going to be implemented in Filament. I think that what you suggest, having a queue of several textures and use those in a round robin fashion will work fine, though the question is how many I should use to be sure that no flickering occurs.
I'm curious to see the impact of calling flushAndWait() will have on my app, but as I'm just started I'm not sure how to predict this. At the moment there is no impact.
flushandwait is really here just for debugging and for shutting down. I think the number of buffers you need will depend on both filament and your app's max latency.
🤐 @pixelflinger how would I go about implementing a round robin solution for this? Normally I would generate a set of textures and change the COLOR0 attachment. I don't see a function which allows me to change the COLOR0 attachment of a RenderTarget. Does this mean I have to create several render targets instead? This probably also means I have to create separate depth buffers as well which seems unnecessary.
Update 1
I've just done an experiment where I create an array of RenderTarget instances, each with a color + depth texture. Then, from my render loop, I change the render target in a round robin fashion.
This doesn't seem to fix the flickering. I've used several different array sizes, 4, 6, 8, 12, 16 and all gave the same result. I'm curious if my round robin implementation is wrong maybe (?).
I've logged the texture IDs of the color attachment that I set as write target and which I use to read/render from. First I make sure that all write textures have been written to. I log Cannot read yet, filling up, writing texture <tex_id>. Then when almost all textures have been written into I start using them to render. I log write: <write_tex_id>, read: <read_tex_id>.
The write_tex_id is the texture ID of the color texture from the render target which is currently used as render target. The read_tex_id is the texture from which I read.
Cannot read yet, filling up, writing texture: 3
Cannot read yet, filling up, writing texture: 4
Cannot read yet, filling up, writing texture: 5
Cannot read yet, filling up, writing texture: 6
Cannot read yet, filling up, writing texture: 7
Cannot read yet, filling up, writing texture: 8
Cannot read yet, filling up, writing texture: 9
Cannot read yet, filling up, writing texture: 10
Cannot read yet, filling up, writing texture: 11
write: 12, read: 3
write: 13, read: 4
write: 14, read: 5
write: 3, read: 6
write: 4, read: 7
write: 5, read: 8
write: 6, read: 9
write: 7, read: 10
write: 8, read: 11
write: 9, read: 12
write: 10, read: 13
write: 11, read: 14
write: 12, read: 3
write: 13, read: 4
write: 14, read: 5
write: 3, read: 6
write: 4, read: 7
write: 5, read: 8
write: 6, read: 9
write: 7, read: 10
write: 8, read: 11
write: 9, read: 12
write: 10, read: 13
write: 11, read: 14
write: 12, read: 3
write: 13, read: 4
write: 14, read: 5
write: 3, read: 6
write: 4, read: 7
write: 5, read: 8
write: 6, read: 9
write: 7, read: 10
write: 8, read: 11
write: 9, read: 12
write: 10, read: 13
write: 11, read: 14
write: 12, read: 3
write: 13, read: 4
write: 14, read: 5
...etc...
Update 2
I'm experimenting a bit. When I disable vsync in my GLFW window: glfwSwapInterval(0), there is no flickering. When I enable vsync again AND remove all my loaded 3D models that I render with Filament, I still have flickering. So this flickering happens when I'm not creating any views, render targets, textures, etc. Only the engine, renderer, swap chain etc.
Maybe some state of my GLFW based GL-context is updated via a call to renderer->beginFrame() or renderer->endFrame().
Interesting: In this video I render the texture into which Filament rendered the scene at the top left of my window. The backgroud is rendered using a very basic setup using my own draw calls and using the GL context that I created. Interestingly, when you see the flickering the full window is drawn using a green color. This green color is the clear color of my default framebuffer. I've also attached a screenshot from the frame which shows the green output.
https://github.com/google/filament/assets/358809/08a6ca4f-acab-4431-8f39-89c68996c024
Update 3
As I just noticed in Update 2, my full window was blinking. This hinted that the blinking was caused by my GLFW render loop and therefore I added a glFinish() call before glfwSwapBuffers(win). This fixes the issue, even without the round robin approach. Event though I'm flushing/waiting the draw calls of my own GL context, I'm not using the flushAndWait() from filament. How does this solve the issue? To be honest I'm not sure what the actual issue is? Is filament doing something with the swapchain?
POTENTIAL FIX:
while (0 == glfwWindowShouldClose(win)) {
glfwMakeContextCurrent(win);
glBindFrameBuffer(GL_FRAMEBUFFER, 0);
glClearColor(0, 0.4, 0, 1);
glViewport(0, 0, width, height);
glClear(GL_COLOR_BUFFER_BIT);
render_with_glfw_context();
render_with_filament();
glfwMakeContextCurrent(win);
glFinish(); /* ⚡ this fixes the blinking */
glfwSwapBuffers();
}
I've looked into this a bit more and someone advised me to see if this issue might be caused by my compositor (xcompmgr). So I ran pkill xcompmgr and started the app several times and the issues seemed to be solved. Even without the potential fix as described in my previous message (see POTENTIAL FIX).
As I thought this was related to the swapchain I looked into the GLX backend implementation which I had looked into several years ago and I noticed that createSwapChain(void* nativeWindow, uint64_t flags = 0) doesn't create a swap chain, it returns the nativeWindow.
I now recall that you have to make use of the the createSwapChain(uint32_t width, uint32_t height, uint64_t flags = 0) variant. This implementation will create a another buffer/swapchain. So the pseudo code for using Filament with a shared OpenGL context is something like:
// pseudo code
win = glfwCreateWindow(width, height, "title", nullptr, nullptr);
glfwMakeContextCurrent(win);
glfwSwapInterval(1);
native_window = gfwGetX11Window(win)
opengl_context = glfwGetGLXContext(win);
engine = Engine::create(backend::Backend::OpenGL, nullptr, opengl_context);
swap_chain = engine->createSwapChain(width, height, 0);
renderer = engine->createRenderer();
while(1) {
if (!renderer->beginFrame(swap_chain)) {
continue;
}
renderer->render(someView);
renderer->endFrame();
}
Now, that I'm using that function I haven't seen any blinking/flickering. As this issue occured randomly I still have to see how it runs after a couple of days but I'm pretty sure this was the issue.