defold
defold copied to clipboard
defold
True Billboards for 3D (label, sprite, particles components)
THE PROBLEM:
At the moment, sprites, labels and particles look flat when viewed in 3D from the side. I'm talking about the "alignment to the camera" that is used for billboards. Here's an example of a video of how it should be: https://user-images.githubusercontent.com/7888071/180055916-35b51fc0-5b0d-4294-8245-a3068aebe7a5.mp4
But that's not the case in Defold right now. :( Look at that miserable particle system - how it flattens out in Defold when rotated in 3D: https://user-images.githubusercontent.com/7888071/180056536-abf4fd57-7423-4ab0-a28c-c5f2d4ec42eb.mp4
The bottom line is this: is it REALLY necessary to pass labels, sprites and particles NOT as billboards? All the games I’ve seen - they all use particles and labels as billboards. It would make sense that this (billboards aligned to the camera) would be the default behaviour for sprites, labels and particles in Defold!
PREFERRED SOLUTION:
Pass in the vertex data the centre of the sprite/label/particle. For example, the vertex declaration for sprites/particles/labels now has the format position + texcoord. Pass in each vertex additionally the centre of the sprite/particle (you know it anyway). With this information, it will be possible to deploy the object frontally in front of the camera in the shader. In fact, this will allow you (or me) to create true and perfect "billboard material". This solution is completely compatible with previous versions of the engine and does not break anything.
What it looks like in my shader:
// we get the central position of object in world coordinates
highp vec4 base = world * vec4(0.0, 0.0, 0.0, 1.0);
// obtain the local offset of the current object vertex to be processed
highp vec4 local_pos = vec4(position.x - base.x, position.y - base.y, 0.0, 0.0);
// putting it all together
gl_Position = (view_proj * base) + (proj * local_pos);
ALTERNATIVE SOLUTION:
Manually rotate all sprites and labels on each frame towards the camera. And you can forget about particle systems in 3D at all. :(
I agree. We want this functionality.
This solution is completely compatible with previous versions of the engine and does not break anything.
The vertex format would be changed, so existing custom shaders would stop working.
It would make sense that this (billboards aligned to the camera) would be the default behaviour for sprites, labels and particles in Defold!
An argument could be made that this should be a setting on the material. For 2D games, it makes little sense to transform the vertices an extra time. We can of course do it, but that will be evaluated during the design phase of the task.
The vertex format would be changed, so existing custom shaders would stop working.
Here's an example.
The beginning of the current shader looks like this:
And it will look like this:
Nothing that would break past custom shaders.
An argument could be made that this should be a setting on the material. For 2D games, it makes little sense to transform the vertices an extra time.
Yes, my idea is that we could make "billboard material" and set it up where we need it - for example for sprites or particles in 3D. So that each user can decide for themselves whether they want true billboards or not, and if they do, they can simply put the appropriate material for their sprites.
so existing custom shaders would stop working.
Even if they did, there have been other cases where we needed to edit custom shaders to update them to work again. Treat it the same as custom manifests, accept that sometimes they will need to be updated as things improve.
If it’s a different vertex stream, it’s not going to break old shaders I think since that attribute won’t be bound if the attribute doesn’t exist. However the vertex data blob will be bigger memory wise even if the attribute is not used, but maybe we can solve that by using different vertex declarations or something. But yes this would be a nice addition to the engine :)
This is possible to do with sprites now since #7508 is done. Not closing this yet since we want the same functionality for the other components as well.
Yes, it allows you to pass the center coordinates for a particular sprite, and with a slight modification of the shader you can get a true billboard for a single sprite. But this still does not solve the problem of true billboard in particle systems.
Yes it’s next on my list, but all the ground work has been laid already so it shouldn’t be too hard to fix
This is possible to do with sprites now since #7508 is done.
Hi. It's great that there is progress in this thread! But unfortunately it's not quite clear how to make a billboard sprite using vertex attributes now? How do you pass the centre of the sprite in local coordinates to the shader, other than writing it by hand in the attribute field? Even when it will be possible to change attributes from the script it will not cancel the need to correct the coordinate from the script, and in the case of sprite movement every frame, which is not much different from rotating the sprite towards the camera from the code.
video (1.6.0 beta): https://github.com/defold/defold/assets/842807/fdbd5661-ec31-44de-bda1-9074e1f35ff6
material:
vertex shader:
// positions are in world space
attribute highp vec4 position;
attribute highp vec3 local;
attribute mediump vec2 texcoord0;
uniform highp mat4 view_proj;
uniform highp mat4 proj;
uniform highp mat4 world;
uniform highp vec4 normal;
uniform highp vec4 cam_pos;
varying mediump vec2 var_texcoord0;
varying highp vec4 var_position;
varying highp vec3 var_normal;
void main()
{
// we get the central position of object in world coordinates
highp vec4 base = vec4(local.xyz, 1.0);
// obtain the local offset of the current object vertex to be processed
highp vec4 local_pos = vec4(position.x - base.x, position.y - base.y, 0.0, 0.0);
// putting it all together
gl_Position = (view_proj * base) + (proj * local_pos);
// gl_Position = view_proj * vec4(position.xyz, 1.0);
var_texcoord0 = texcoord0;
var_position = vec4(position.xyz, 1.0);
var_normal = normalize((cam_pos).xyz);
// var_normal = normalize(normal.xyz);
}
As far as I understand vertex attribute with semantic type is "position" and local space passes an abstract coordinate in the range -.5 .5 to the shader. Maybe there are some other options for customisation?
Yeah I think I spoke to soon regarding this feature. We have had discussions on how to add the remaining bits into the vertex format so we can close this once and for all, sorry for the confusion!
Looks like I finally made billboard sprites and particle_fx with the means available now. Without having to pass data to them in runtime. Changed the shader a bit. Demo example: https://dragosha.com/defold/tiny_world/ LMB - move map RMB - rotate map Here we have sprite trees and particles that are facing the camera.
Here is the vertex shader in the case of particles:
uniform highp mat4 view_proj;
uniform highp mat4 proj;
// positions are in world space
attribute highp vec3 local_position;
attribute highp vec4 position;
attribute mediump vec2 texcoord0;
attribute lowp vec4 color;
varying mediump vec2 var_texcoord0;
varying lowp vec4 var_color;
void main()
{
highp vec4 base = vec4(position.x - local_position.x, position.y - local_position.y, position.z - local_position.z, 1.0);
highp vec4 local_pos = vec4(position.x - base.x, position.y - base.y, 0.0, 0.0);
gl_Position = (view_proj * base) + (proj * local_pos);
var_texcoord0 = texcoord0;
var_color = vec4(color.rgb * color.a, color.a);
}
And material settings:
However, some questions appeared. In the case of particles, the shader works correctly both in the editor (I'm looking at 1.6.0 beta) and in runtime.
Editor:
Runtime:
But in the case of sprite shader:
Editor (look at trees):
Runtime (look at trees, billboarding not applied):
I've fixed this by passing to the shader additional vertex attribute, it's size of sprite in pixels.
attribute highp vec3 size;
...
highp vec4 base = vec4(position.x - local_position.x * size.x, position.y - local_position.y, position.z - local_position.z, 1.0);
or even
highp vec4 base = vec4(position.x - local_position.x * size.x, position.y - local_position.y * size.y, position.z - local_position.z, 1.0);
if I want to see strict facing the camera.
Material setup:
and sprite setup, where added manual size of sprite in pixels:
It works correct in runtime (you can see this version of shader in the HTML5 demo).
But in the editor it looks a bit screw:
So, the question is, how to fix that? :)
Great work! Good question about the sprites. Particle vertices are generated the exact same way for editor and engine, but sprite vertices are not. Let me get back to you on that. One of them should produce the correct result, just need to establish which one of them we want :)
I think this code:
highp vec4 base = vec4(position.x - local_position.x, position.y - local_position.y, position.z - local_position.z, 1.0);
highp vec4 local_pos = vec4(position.x - base.x, position.y - base.y, 0.0, 0.0);
gl_Position = (view_proj * base) + (proj * local_pos);
can be replaced with a single line:
gl_Position = (view_proj * vec4(position.xyz - local_position.xyz, 1.0)) + (proj * vec4(local_position.xy, 0.0, 0.0));
But I actually have a question about the meaning of the attributes.
WHAT DO THIS MEAN? Judging from the way it works, we can assume that these are the coordinates of the vertex of the particle/sprite in local space, where the center of the particle/sprite is at zero. Moreover, they are already has been transformed through world and view matrices?
Judging from the way it works, we can assume that these are the coordinates of the vertex of the particle/sprite in local space, where the center of the particle/sprite is at zero.
Moreover, they are already has been transformed through world and view matrices?
They have not been transformed by any matrices, they are expressed in pixel coordinates relative to zero, where the center of the sprite / particle is origo. I have found an inconsistency in the engine that produced coordinates in -0.5 and 0.5 for sprites, where it in reality should be based on the size of the sprite. Working on a PR now and then I think we should be able to close this issue yes?
@Dragosha: when this build is done, could you try this? https://github.com/defold/defold/actions/runs/6262497243/job/17007355931 It has a fix for the local position
@Dragosha d.defold.com/archive/dev/143728d6755653a79bac9b151d4d19be0631cf8d/dev/editor2/Defold-x86_64-win32.zip
Yay! It works!
Is it possible to celebrate a great accomplishment? And open a bottle of cognac in honor of the occasion?
I think it makes sense to include the true billboard shader as the default shader for particles and sprites. I don't see any reason why people would need to use the old shaders, since the new ones are fully compatible.
@Jhonnyg @britzl @AGulev what do you think?
I think we should leave it as it is. There are cases when sprites are used as planes in building a 3D scene, and they should keep their angles. My game foggy fox is full of this. For example, a sprite as a glare. Besides, it is not always necessary to rotate the sprite strictly into the camera on all axes. For example, sometimes one axis is enough. As for particles, maybe indeed a shader change would come in useful.
But again. Local position is an extra attribute that is not needed for 2d games.
Yeah, I think we won't make this the default behaviour for the builtin shaders but instead I think it would make sense to add a billboard material + vertex shader to the builtins with the extra attribute. Since we want to add a center semantic type to the attributes those bits will probably be added after that feature is done.
It seems that the calculations do not take the sprite scale, no matter if it is set for a component or an game object.
The scale is 0.5:
1 to 1:
@Dragosha: when this build is done, could you try this? defold/defold/actions/runs/6262497243/job/17007355931 It has a fix for the local position
@Jhonnyg Local position coordinates are based on sprite size in editor and runtime, but if sprite has overridden any vertex attribute then coordinates are between -0.5 and 0.5 ONLY in runtime
