sokol
sokol copied to clipboard
floooh
Automatic mipmap generation
Hi, is there any plan to add automatic mipmap generation? If not, would you be ok with a pull request adding it? It looks like all your target API's support it:
https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/generateMipmap https://www.khronos.org/registry/OpenGL-Refpages/gl4/html/glGenerateMipmap.xhtml https://developer.apple.com/documentation/metal/mtlblitcommandencoder/1400748-generatemipmapsfortexture https://docs.microsoft.com/en-us/windows/desktop/api/d3d11/nf-d3d11-id3d11devicecontext-generatemips
I haven't planned it so far, but I think it's a good addition. Implementing it may be a bit trickier than expected because of the sokol validation layer (at least need to check what's the common subset of texture pixel formats which allow mipmap generation across the 3D APIs, and the image creation stuff which needs to know that mipmaps might be generated).
I wonder if it makes sense to not make it an explicit call, but only set a flag in sg_image_desc for sg_create_image(), and the mipmap generation takes place when needed:
- immutable textures would only be provided the data for the top-level mipmap, and the mipmap generation happens once during
sg_create_image(), the data cannot change anyway later - dynamic textures would generate mipmaps whenever the texture data is updated via
sg_update_image()(the update would only provide the data for the first mip level)
At first glance I think I like this automatic generation better, but there may be API-specific caveats with things like the separate Metal BlitCommandEncoder (I haven't used this yet).
I agree, it definitely makes sense to have automatic generation, and it should also happen when an image is rendered into (that's my use case). It probably wouldn't hurt to allow manual generation as well, right? I'm porting one of my image effect shaders over from Unity, which it has both, you can set a RenderTexture as autoGenerateMipmaps=true or call GenerateMips() on it. I've never had a need for GenerateMips(), but I imagine it could be useful when there are multiple draw calls on a render texture, and you only want to generate mipmaps at the end.
How about this -- I can try to create a basic manual version sg_generate_mips() (or similar) for DirectX/OpenGL/WebGL (no Mac to test on). If that works ok, automatic generation could be a next step that just calls the manual version.
Yes, I agree, doing the explicit call first is better to check out whether everything works as intended, after that we can decide wether automatic generation still makes sense. My only concern is that the validation stuff will become more complex (or less "watertight").
For instance currently you need to tell sg_create_image() how many mipmaps you have, and the initial data size must match the expected size for all mipmaps, and in addition the texture filtering must be compatible. You're going to need to figure something out there ;)
PS: when you have the DX/GL case working I can look into the Metal version.
Yes, you're right, it could become more complex for the user. In addition, WebGL1 (and maybe GLES2?) only supports generating mipmaps for power-of-two textures, so that would need to be checked too. The tough part will be counting mipmaps and checking pixel formats, though.
We could add an enum sg_mipmap_generation { SG_MIPMAPGENERATION_NONE, SG_MIPMAPGENERATION_ON_DEMAND, SG_MIPMAPGENERATION_AUTO } and use it in sg_image_desc as mipmap_generation (or mipmap_gen? all these names are flexible, feel free to change them). If it's not NONE:
- If provided num_mipmaps > 0: supply num_mipmaps based on size
- If provided num_mipmaps > 1: validate that it matches the required number
- Validate texture filtering, pixel format, and power-of-two textures on appropriate platforms
The enum value is stored in _sg_image.mipmap_generation, and it can be quickly validated when sg_generate_mipmaps() is called. It can also (eventually) be checked in auto mode for sg_update_image and when used as a render texture.
What do you think?
Yes sounds good, but in the end we need to check how it "feels" in real code ;) Some minor naming proposals:
SG_MIPMAPGENERATION_ON_DEMAND -> SG_MIPMAPGENERATION_EXPLICIT? (meaning that sg_generate_mipmaps() needs to be called explicitely, while _AUTO mode would do the automatic "under-the-hood" generation? (not sure yet whether we actually need the explicit generation apart from the initial implementation...?)
...if we decide later that we don't even need the explicit call, the enum could be reduced to a bool, since we only have either "generate_mipmaps" on or off in sg_image_desc.
The WebGL1 limitation for 2^N textures is a bit of a bummer :/
This I have a bit of trouble understanding:
- If provided num_mipmaps > 0: supply num_mipmaps based on size
- If provided num_mipmaps > 1: validate that it matches the required number
...but maybe it makes more sense to discuss such details over actual code ;)
SG_MIPMAPGENERATION_ON_DEMAND -> SG_MIPMAPGENERATION_EXPLICIT
Yes, better!
not sure yet whether we actually need the explicit generation apart from the initial implementation...?
Well, what about mipmapped render textures? I suppose you could wait until the end of the pass to auto generate mipmaps? But you may still want finer control over when they are generated... I mean, it's a pretty obscure use case for sure 😄
The WebGL1 limitation for 2^N textures is a bit of a bummer :/
Yeah, I keep running into new and exciting WebGL1 limitations 😆 I wish Apple would finally allow WebGL2 on Safari.
This I have a bit of trouble understanding:
I just mean that num_mipmaps would be automatically calculated or, if supplied, it would be verified. Maybe it would be better to just have it error if num_mipmaps is nonzero with SG_MIPMAPGENERATION_* enabled.
Well, what about mipmapped render textures? I suppose you could wait until the end of the pass to auto generate mipmaps? But you may still want finer control over when they are generated... I mean, it's a pretty obscure use case for sure
Yes, the mipmap generation would happen in sg_end_pass(), similar to the MSAA resolve. You wouldn't be able to use the rendered image as a texture before sg_end_pass() anyway.
PS: ...and the sg_generate_mipmaps() call would need to happen after the sg_end_pass() (one open question would be: does it need to happen inside one of the following sg_begin_pass()/sg_end_pass() pairs, or outside any pass? Such things don't need to be decided with the "automatic" generation :)
Hey, I just wanted to give a quick update. I finally found some time to work on this, and I've implemented a version that works with OpenGL and WebGL. After looking over everything again, I have to say I agree with you on the fully automated route! It doesn't sacrifice much flexibility, and it makes the API much easier. My implementation is fully automatic, it just adds a bool autogen_mipmaps field to the sg_image_desc struct. Mipmaps are auto-generated when autogen_mipmaps is true and:
- when an immutable, non-render_target image is created
- when an image is updated
- after a pass (in _sg_end_pass) if the image is in a pass's color_atts
I'm still testing it while I convert my shaders, I will try to get the DirectX side done and make a pull request soon.
Any update on mipmap generation? I've been waiting for this feature for quite a while!
I'll probably pick up this PR after I finished on the pixel-format modernization (but I'm feeling an emulator-itch coming again, so it may not be immediately after).
We'll also have to see how fast this is on the various APIs. It's been a while, but I remember that I stumbled over some GL drivers where mipmap generation was just too slow to do every frame.
Well it could be nice to at least have an interface that calls glGenerateMipmaps when we initialize the contents of the texture. For now I hacked sokol a little bit and added the glGenerateMipmaps after glTexImage2D call, but I would like to have a prettier implementation.
I made the changes for D3D11 and OpenGL at my fork here: https://github.com/russpowers/sokol
It's been a while, and I don't think I properly tested things at the time, although it should be fairly close. I'll need to take another look at it before submitting a PR.
I currently use workaround by generating mip-maps by myself while waiting for official implementation. Probably has artifacts and isn't as fast etc. But at least works for all platforms. (I currently only support RGBA8 and R8 formats as I don't need others). Usage is as simple as calling sg_make_image_with_mipmaps instead of sg_make_image.
If interested see: https://github.com/Deins/sokol/tree/soft_gen_mipmaps
Deins's solution seems to like to segfault on my machine, sometimes at the call site other times in the graphics driver.
EDIT: seems to be the second time I call it, if I use a differently sized image than the first time?
EDIT2: Removing static from buffers seems to fix it, interesting.
Oh, sorry, yes the static has a bug. It has to be removed or the array has to be set to 0 each time, or at the end where its contents gets freed. I think I added the static being paranoid about stack overflow, but I don't think it should be an issue, so just removing it seems to be best.
Still be warned, I used that code quite some time ago and it was for really small experiment, so it isn't tested too well, but hopefully that's it.
FYI, I cherrypicked @Deins into my own fork and fixed a few minor things that had changed in Sokol:
https://github.com/Srekel/sokol/commits/master
Seems to work fine for me :)
I'd like to bump this issue as having official support would be nice. In the meantime, I took the commit from @Srekel and fixed some issues and added support for texture arrays
- The buffer for each mip was too big (the size of a level above).
- The buffer was allocated before doing a check if it's needed.
There's also a "Chanell" misspell lol but I left it. Making one large buffer for all mips levels and passing the same pointer with offsets might be a good idea too, but I didn't do it here.
EDIT: At the bottom I posted a version with one allocation
From edb2c7e9fd2815504b6d7209dbb76370c576dff6 Mon Sep 17 00:00:00 2001
From: pseregiet <[email protected]>
Date: Fri, 31 Dec 2021 00:43:31 +0100
Subject: [PATCH] sg_make_image_with_mipmaps
---
sokol_gfx.h | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 75 insertions(+)
diff --git a/sokol_gfx.h b/sokol_gfx.h
index e084e24..d7f5be5 100644
--- a/sokol_gfx.h
+++ b/sokol_gfx.h
@@ -2277,6 +2277,7 @@ SOKOL_GFX_API_DECL void sg_pop_debug_group(void);
/* resource creation, destruction and updating */
SOKOL_GFX_API_DECL sg_buffer sg_make_buffer(const sg_buffer_desc* desc);
SOKOL_GFX_API_DECL sg_image sg_make_image(const sg_image_desc* desc);
+SOKOL_GFX_API_DECL sg_image sg_make_image_with_mipmaps(const sg_image_desc* desc);
SOKOL_GFX_API_DECL sg_shader sg_make_shader(const sg_shader_desc* desc);
SOKOL_GFX_API_DECL sg_pipeline sg_make_pipeline(const sg_pipeline_desc* desc);
SOKOL_GFX_API_DECL sg_pass sg_make_pass(const sg_pass_desc* desc);
@@ -15164,6 +15165,80 @@ SOKOL_API_IMPL sg_image sg_make_image(const sg_image_desc* desc) {
return img_id;
}
+SOKOL_API_IMPL sg_image sg_make_image_with_mipmaps(const sg_image_desc* desc_)
+{
+ sg_image_desc desc = *desc_;
+ SOKOL_ASSERT(desc.pixel_format == SG_PIXELFORMAT_RGBA8
+ || desc.pixel_format == SG_PIXELFORMAT_BGRA8
+ || desc.pixel_format == SG_PIXELFORMAT_R8);
+
+ unsigned pixel_size = _sg_pixelformat_bytesize(desc.pixel_format);
+ unsigned char* buffers[SG_CUBEFACE_NUM][SG_MAX_MIPMAPS] = {0}; // TODO: better allocation
+
+ for (int cube_face = 0; cube_face < SG_CUBEFACE_NUM; ++cube_face)
+ {
+ int target_width = desc.width;
+ int target_height = desc.height;
+ int dst_height = target_height;
+ if (desc.num_slices)
+ dst_height *= desc.num_slices;
+
+ for (int level = 1; level < SG_MAX_MIPMAPS; ++level)
+ {
+ unsigned char* source = (unsigned char*)desc.data.subimage[cube_face][level - 1].ptr;
+ if (!source) break;
+ int source_width = target_width;
+ int source_height = target_height;
+ target_width /= 2;
+ target_height /= 2;
+ if (target_width < 1 && target_height < 1) break;
+ if (target_width < 1) target_width= 1;
+ if (target_height < 1) target_height = 1;
+
+ dst_height /= 2;
+ unsigned img_size = target_width * dst_height * pixel_size;
+ unsigned char* target = (unsigned char*)SOKOL_MALLOC(img_size);
+ buffers[cube_face][level] = target;
+
+ for (int slice = 0; slice < desc.num_slices; ++slice) {
+ for (int x = 0; x < target_width; ++x)
+ {
+ for (int y = 0; y < target_height; ++y)
+ {
+ uint16_t colors[8] = { 0 };
+ for (int chanell = 0; chanell < pixel_size; ++chanell)
+ {
+ int color = 0;
+ int sx = x * 2;
+ int sy = y * 2;
+ color += source[source_width * pixel_size * sx + sy * pixel_size + chanell];
+ color += source[source_width * pixel_size * (sx + 1) + sy * pixel_size + chanell];
+ color += source[source_width * pixel_size * (sx + 1) + (sy + 1) * pixel_size + chanell];
+ color += source[source_width * pixel_size * sx + (sy + 1) * pixel_size + chanell];
+ color /= 4;
+ target[target_width * pixel_size * (x) + (y) * pixel_size + chanell] = (uint8_t)color;
+ }
+ }
+ }
+
+ source += (source_width * source_height * pixel_size);
+ target += (target_width * target_height * pixel_size);
+ }
+ desc.data.subimage[cube_face][level].ptr = buffers[cube_face][level];
+ desc.data.subimage[cube_face][level].size = img_size;
+ if (desc.num_mipmaps <= level) desc.num_mipmaps = level + 1;
+ }
+ }
+
+ sg_image img = sg_make_image(&desc);
+ for (int cube_face = 0; cube_face < SG_CUBEFACE_NUM; ++cube_face) {
+ for (int i = 0; i < SG_MAX_MIPMAPS; ++i) {
+ SOKOL_FREE(buffers[cube_face][i]);
+ }
+ }
+ return img;
+}
+
SOKOL_API_IMPL sg_shader sg_make_shader(const sg_shader_desc* desc) {
SOKOL_ASSERT(_sg.valid);
SOKOL_ASSERT(desc);
--
2.34.1
with one malloc:
From c80250e0f7d83367afbbb57446f35565cef3c018 Mon Sep 17 00:00:00 2001
From: pseregiet <[email protected]>
Date: Fri, 31 Dec 2021 00:43:31 +0100
Subject: [PATCH] sg_make_image_with_mipmaps
---
sokol_gfx.h | 97 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 97 insertions(+)
diff --git a/sokol_gfx.h b/sokol_gfx.h
index e084e24..72c5d62 100644
--- a/sokol_gfx.h
+++ b/sokol_gfx.h
@@ -2277,6 +2277,7 @@ SOKOL_GFX_API_DECL void sg_pop_debug_group(void);
/* resource creation, destruction and updating */
SOKOL_GFX_API_DECL sg_buffer sg_make_buffer(const sg_buffer_desc* desc);
SOKOL_GFX_API_DECL sg_image sg_make_image(const sg_image_desc* desc);
+SOKOL_GFX_API_DECL sg_image sg_make_image_with_mipmaps(const sg_image_desc* desc);
SOKOL_GFX_API_DECL sg_shader sg_make_shader(const sg_shader_desc* desc);
SOKOL_GFX_API_DECL sg_pipeline sg_make_pipeline(const sg_pipeline_desc* desc);
SOKOL_GFX_API_DECL sg_pass sg_make_pass(const sg_pass_desc* desc);
@@ -15164,6 +15165,102 @@ SOKOL_API_IMPL sg_image sg_make_image(const sg_image_desc* desc) {
return img_id;
}
+SOKOL_API_IMPL sg_image sg_make_image_with_mipmaps(const sg_image_desc* desc_)
+{
+ sg_image_desc desc = *desc_;
+ SOKOL_ASSERT(desc.pixel_format == SG_PIXELFORMAT_RGBA8
+ || desc.pixel_format == SG_PIXELFORMAT_BGRA8
+ || desc.pixel_format == SG_PIXELFORMAT_R8);
+
+ unsigned pixel_size = _sg_pixelformat_bytesize(desc.pixel_format);
+ int w = desc.width;
+ int h = desc.height * desc.num_slices;
+ int total_size = 0;
+ for (int level = 1; level < SG_MAX_MIPMAPS; ++level) {
+ w /=2;
+ h /=2;
+
+ if (w < 1 && h < 1)
+ break;
+
+ total_size += (w * h * pixel_size);
+ }
+
+ int cube_faces = 0;
+ for (; cube_faces < SG_CUBEFACE_NUM; ++cube_faces) {
+ if (!desc.data.subimage[cube_faces][0].ptr)
+ break;
+ }
+
+ total_size *= (cube_faces+1);
+ unsigned char *big_target = SOKOL_MALLOC(total_size);
+ unsigned char *target = big_target;
+
+ for (int cube_face = 0; cube_face < cube_faces; ++cube_face)
+ {
+ int target_width = desc.width;
+ int target_height = desc.height;
+ int dst_height = target_height * desc.num_slices;
+
+ for (int level = 1; level < SG_MAX_MIPMAPS; ++level) {
+ unsigned char* source = (unsigned char*)desc.data.subimage[cube_face][level - 1].ptr;
+ if (!source)
+ break;
+
+ int source_width = target_width;
+ int source_height = target_height;
+ target_width /= 2;
+ target_height /= 2;
+ if (target_width < 1 && target_height < 1)
+ break;
+
+ if (target_width < 1)
+ target_width = 1;
+
+ if (target_height < 1)
+ target_height = 1;
+
+ dst_height /= 2;
+ unsigned img_size = target_width * dst_height * pixel_size;
+ unsigned char *miptarget = target;
+
+ for (int slice = 0; slice < desc.num_slices; ++slice) {
+ for (int x = 0; x < target_width; ++x)
+ {
+ for (int y = 0; y < target_height; ++y)
+ {
+ uint16_t colors[8] = { 0 };
+ for (int chanell = 0; chanell < pixel_size; ++chanell)
+ {
+ int color = 0;
+ int sx = x * 2;
+ int sy = y * 2;
+ color += source[source_width * pixel_size * sx + sy * pixel_size + chanell];
+ color += source[source_width * pixel_size * (sx + 1) + sy * pixel_size + chanell];
+ color += source[source_width * pixel_size * (sx + 1) + (sy + 1) * pixel_size + chanell];
+ color += source[source_width * pixel_size * sx + (sy + 1) * pixel_size + chanell];
+ color /= 4;
+ miptarget[target_width * pixel_size * (x) + (y) * pixel_size + chanell] = (uint8_t)color;
+ }
+ }
+ }
+
+ source += (source_width * source_height * pixel_size);
+ miptarget += (target_width * target_height * pixel_size);
+ }
+ desc.data.subimage[cube_face][level].ptr = target;
+ desc.data.subimage[cube_face][level].size = img_size;
+ target += img_size;
+ if (desc.num_mipmaps <= level)
+ desc.num_mipmaps = level + 1;
+ }
+ }
+
+ sg_image img = sg_make_image(&desc);
+ SOKOL_FREE(big_target);
+ return img;
+}
+
SOKOL_API_IMPL sg_shader sg_make_shader(const sg_shader_desc* desc) {
SOKOL_ASSERT(_sg.valid);
SOKOL_ASSERT(desc);
--
2.34.1
It probably won't go into sokol_gfx.h itself, but maybe into a utility header (there would need to be two separate code paths though, one that runs on the CPU and which takes an sg_image_data struct with a single toplevel surface and populates the mipmap cascade, and another one which takes a render target texture as input populates the mipmap cascade on the GPU through one render pass per mipmap surface.
What's the thinking behind writing two implementations (CPU side & GPU side mipmap generation)? Why not only implement GPU — works both for new images and existing ones.
To generate mipmaps on the GPU you would need to declare the image as render target, and kick off one render pass per mipmap. In such situations it might actually be simpler to just run the mipmap generation on the CPU before even creating the image object, while for images that are offscreen render targets in the first place the GPU rendering method would be better.
The situation for builtin mipmap generation support seems to be:
- D3D11: https://learn.microsoft.com/en-us/windows/win32/api/d3d11/nf-d3d11-id3d11devicecontext-generatemips
- Metal: https://developer.apple.com/documentation/metal/textures/generating_mipmap_data
- GL/WebGL2: glGenerateMipmaps() (however in the past I stumbled over some extremely slow mipmap generation implementations in GL, literally taking hundreds of milliseconds)
...however most modern 3D APIs dropped builtin support for generating mipmaps (D3D12, Vulkan and WebGPU), best option there seems to be to use compute shaders.
Eventually I want compute shader support in sokol-gfx, and when that's in place, the most straightforward solution would be a utility header with a unified mipmap generation method running in a compute shader. This means that WebGL2 can't be supported though.