Rebuilding shaders broken on Windows

[Synxis]

I'm opening this issue just to notify that rebuilding example shaders on Windows is far from working out of the box. I can get it to work but usually after a bit of time fixing problems (environment is MSYS2). I do a make in the examples folder, maybe this is not the right command ?

Anyway:

• echo to a file does not work (for example -@echo extern const uint8_t* $(basename $(<))_pssl;>> $(@)), so the last 2 lines about pssl are never added. For my fork I edit all the files at hand for now...

• tons of weird directories are created (example: 13-stencil/.....examplesruntimeshadersdx11, 05-instancing/.....buildshadersspirv, etc...).

• the build and final directories are not automatically created (i.e., the dirs target does nothing). Or, more realistically, it does not work as intended and creates the weird folders of the previous bullet point.

• the compilation fails for some complex shaders:

   [cs_update.sc]
   cpp: "../../scripts//../src/bgfx_compute.sh", line 58: Fatal: Work buffer overflow
    from file cs_update.sc, line 83.
   cpp: "../../scripts//../src/bgfx_compute.sh", line 58: Fatal: Work buffer overflow
    from file cs_update.sc, line 83.
   cpp: "../../scripts//../src/bgfx_compute.sh", line 58: Fatal: Work buffer overflow
    from file cs_update.sc, line 83.
   cpp: "../../scripts//../src/bgfx_compute.sh", line 58: Fatal: Work buffer overflow
    from file cs_update.sc, line 83.
   cpp: "../../scripts//../src/bgfx_compute.sh", line 58: Fatal: Work buffer overflow
    from file cs_update.sc, line 83.
   cpp: "../../scripts//../src/bgfx_compute.sh", line 58: Fatal: Work buffer overflow
    from file cs_update.sc, line 83.
   cpp: "../../scripts//../src/bgfx_compute.sh", line 58: Fatal: Work buffer overflow
    from file cs_update.sc, line 83.
   cpp: "../../scripts//../src/bgfx_compute.sh", line 58: Error: Inside #ifdef block at end of input, depth = 3
    from file cs_update.sc, line 83.
   Shader entry point 'void main()' is not found.
  

  A quick fix is to increase the buffer sizes in fcpp (I put 4096 for NBUFF and NWORK)

• not all shaders in runtime/shaders/ were touched by the rebuild. Are some shaders obsolete but still present in the repo ? These shaders are:

  • runtime/shaders/glsl/fs_shadowmaps_color_lighting.bin
  • runtime/shaders/glsl/fs_shadowmaps_color_lighting_omni.bin
  • runtime/shaders/glsl/fs_shadowmaps_color_lighting_pfc.bin
  • runtime/shaders/glsl/fs_shadowmaps_depth.bin

That's all I saw, after that I was able to rebuild shaders completely.

[bkaradzic]

This is happening in your fork or master?

[Synxis]

Both my fork and master

[bkaradzic]

I did rebuild last night to check this, deleted all built shaders and no issues: https://github.com/bkaradzic/bgfx/commit/610055fbb04f83b4d1284d46ba8c6abaacf88ba6

Which compiler you're using for building shaderc?

[Synxis]

Visual Studio 2015 on Win10. But most of the problems are from the makefile. You rebuilt the shaders on linux ?

[bkaradzic]

I rebuild them on windows because D3DCompiler works only there... Issue you pointed out above is not due to makefile.

[Synxis]

Issues 1 and 2 clearly are from the makefile. Issue 3, I don't know why it seems to work properly on your system and not on mine. Note that I just do make rebuild inside the examples directory.

[bkaradzic]

1, 2 are because you have multiple shells in your path. Most common issue is sh that comes from Git. Just remove git/bin from your path and it will find GnuWin32 one.

[Synxis]

I do not have the git shell in MSYS.

I found the issue for 1. In the following line -@echo extern const uint8_t* $(basename $(<))_pssl;>> $(@) (in shader-embedded.mk), the ; is interpreted as a command separator, hence the redirection does not apply... Adding a backslash to escape the semicolon solves it.

I also found the issue for 2. In tools.mk, there is CMD_MKDIR=cmd /C "if not exist "$(subst /,\,$(1))" mkdir "$(subst /,\,$(1))"" for Windows. It turns out replacing slash by antislashes ends up escaping characters... Changing $(subst /,\,$(1)) by $(subst /,\\,$(1)) solves the problem.

Btw, compiling shaders for Vulkan outputs a lot of debug info of the form:

[fs_particle.sc]
2401: BgfxSampler2D -1
{
                 m_sampler -1
                 m_texture -1
}
499: $Global -1
{
                 u_viewRect 0
                 u_viewTexel 16
                 u_view 32
                 u_invView 96
                 u_proj 160
                 u_invProj 224
                 u_viewProj 288
                 u_invViewProj 352
                 u_model 416
                 u_modelView 2464
                 u_modelViewProj 2528
                 u_alphaRef4 2592
}

[dlannan]

I dont know if this is related, but I cannot run many of the example exe's after building in 2015 either in x64 or win32. Only these examples work: 00, 02, 17, 20, 25 and 29 However, if I run them within the Visual Studio IDE, they all work correctly?

[dlannan]

Arghh! slap me stupid. Read the effin docs. Soz.

[Synxis]

Normally you don't have to recompile the shaders. This bug report is for when you have to (for example modification of the binary format). @bkaradzic in which environment you were running the makefiles ? And which compiler ?

[EvilTrev]

I just hit this same snag with NWORK not being large enough to parse bgfx_compute.sh due to its massive macros. For my work environment (macOS, Xcode) the version of shaderc I built ended up with an NWORK size of 512 due to the way it is defined in cppdef.h. The first define is for 2048, but that doesn't seem to get defined for this build and the default is then 512 (as defined lower in the same file). I altered that value and was able to get things compiling again.

[bkaradzic]

If NWORK is an issue you might not be using provided build system to build shaderc: https://github.com/bkaradzic/bgfx/blob/4f878bf473e91e373046fe38871325cd8c992b43/scripts/shaderc.lua#L519-L521