WIP: embed media files

[illwieckz]

Work in progress: embed media files from daemon_src.dpkdir.

I deeply reworked the GLSL include generator to make it reusable, I also renamed DaemonBuildInfo.cmake as DaemonGeneratedSource.cmake and put everything about generating sources there, and revamped it a lot.

It doesn't work yet because:

• The current code uses strings and then files are truncated on first 0, but the sound.wav file contains some bytes like that and this is valid.
• Files aren't listed, so there is no way the engine.shader file gets parsed.

[illwieckz]

So it now loads the whole file.

For a funny reason, it works because since the unvanquished dpk lists those names, the engine looks for them, and then the code to read them works. Of course we should be able to list them even if they aren't in a dpk already.

[illwieckz]

It almost works but…

• This error:

Warn: libpng error: PNG file corrupted by ASCII conversion 
Warn: PNG image 'gfx/2d/bigchars.png' has first exception handler called [libpng v.'1.6.50'] 

• The files are embedded in all engines (including the server and the tty client).
• I guess the server now requires the client to have that pak loaded just because the engine loaded it.

[illwieckz]

This error

Now fixed, there was a source beautifier stripping \r, now that we not only embed GLSL files, we better drop that. Also we better beautify the sources in the repository directly.

[slipher]

Now fixed, there was a source beautifier stripping \r, now that we not only embed GLSL files, we better drop that.

Don't drop it. Add an option to specify text or binary mode.

[illwieckz]

Hmm, right now there is something corrupting something, the engine is slow, and the console has some things printed everyframe, actually it prints some PK string so it looks like the console is attempting to print the content of a zip every frame… that's very weird.

[illwieckz]

Don't drop it. Add an option to specify text or binary mode.

I added an option.

[illwieckz]

I now should have filtered out the builtin pak name from the list of required paks sent to clients.

I still don't know from where comes that weird bug slowing down the engine and polluting the console…

[illwieckz]

So I stop there for now, for people around, don't hesitate to look at it, especially for tracking bugs! 😁️

[slipher]

Hmm, in #71 you said we "don't have to think about some generic dpk embedding mechanism", but it looks like you have implemented precisely that. I was thinking we would avoid that and just use the byte arrays generated by the existing embed mechanism directly where needed. For example if we wanted the "bigchars" image, we could add that to the builtin images: create a *consoleFallbackFont image which would upload in the embedded byte array as an image. No need to make it a VFS file. Similarly, for q3shaders, we could just have a string in the source code and feed it into the shader parsing function.

[illwieckz]

Hmm, in #71 you said we "don't have to think about some generic dpk embedding mechanism", but it looks like you have implemented precisely that.

Yes, life is funny. 😄

I started wiring the VFS to test the loading of files, because in fact it's also a good way to make sure the path is exposed and that all the decoding functions are used. For example the PNG image has to be decoded by PNG, etc.

I don't mind the implementation, what I wanted to do there is to plumb the actual file embedding, which seems to now work.

I don't know how to properly make those files usable once they are embedded, and that's a task anyone can pick up.

[illwieckz]

Another thing funny is that I tried to create the smallest null.wav file possible, but then I get this error:

Warn: Unhandled exception (St12length_error): cannot create std::vector larger than max_size() 

My “smaller file” is using such format: signed 16-bit mono 8000 Hz, likely one sample only.

[illwieckz]

Ok, I had to keep enough samples, something like 0.3ms, the file now weights 90 bytes instead of 20KB, here is the generated file:

const unsigned char sound_null_wav[] = {
0x52,0x49,0x46,0x46,0x52,0x00,0x00,0x00,0x57,0x41,0x56,0x45,0x66,0x6d,0x74,
0x20,0x10,0x00,0x00,0x00,0x01,0x00,0x01,0x00,0x40,0x1f,0x00,0x00,0x80,0x3e,
0x00,0x00,0x02,0x00,0x10,0x00,0x64,0x61,0x74,0x61,0x2e,0x00,0x00,0x00,0x00,
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,
0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,};

Edit: I also made sure it was truly zeroed.

[illwieckz]

Actually, when emptying the embedded metadata string we cant remove™ from the flac encoder, I can get a 66 bytes file:

const unsigned char sound_null_flac[] = {
0x66,0x4c,0x61,0x43,0x00,0x00,0x00,0x22,0x10,0x00,0x10,0x00,0x00,0x00,0x0c,
0x00,0x00,0x0c,0x01,0xf4,0x00,0xf0,0x00,0x00,0x00,0x17,0xd8,0x98,0x50,0x4a,
0x72,0x2b,0xff,0x15,0x24,0x13,0x4c,0x6a,0xb6,0xa5,0xea,0xa5,0x84,0x00,0x00,
0x08,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0xff,0xf8,0x64,0x08,0x00,0x16,
0x81,0x00,0x00,0x00,0xc6,0xa3,};

Edit: Well, I'm stupid, the engine doesn't know how to decode flac…

[illwieckz]

I embedded the unifont and NUKED the bigchars.

[illwieckz]

I added a mechanism for any part of the engine to register a builtin pak.

Since the builtin pak registration is done in the code using the builtin pak, the FileSystem code doesn't hardcode any builtin pak, and this made possible to only embed the builtin pak in the client.

Two effects:

• we can have different builtin paks per binaries. For example the vulkan renderer may embed some files, while the OpenGL one would embed others…
• we can now embed more thane one builtin pak. For example we can imagine a client builtin pak, a server builtin pak, and a common builtin pak.
• we can have build-variants of builtin paks. For example a test builtin pak with test files can now only be embedded when guarded behind a build option.

[illwieckz]

So the last problem remaining is that weird bug of the engine being slow and printing garbage PK strings when starting a map.

[illwieckz]

I added a special empty *daemon pakpath, so one can run the engine without any pakpath by doing:

./daemon -set fs_basepath '*daemon'

This makes possible to drop the daemon dpk.

[illwieckz]

I now discover that the daemon-tty client attempts to load the unifont…

[illwieckz]

Hmm, there is something weird with the fact the unifont is loaded by the engine, but when loading the game code it doesn't.

[illwieckz]

Hmm, there is something weird with the fact the unifont is loaded by the engine, but when loading the game code it doesn't.

It works with the client, not with the tty client.

[illwieckz]

Hmm, there is something weird with the fact the unifont is loaded by the engine, but when loading the game code it doesn't.

It works with the client, not with the tty client.

That's because in the tty client RE_RegisterFont() is a stub always returning nullptr, we better comment out the code calling it. Now done.

[illwieckz]

It looks like I only have one bug remaining, that weird printing of PK strings when loading a map with the client, bug which is also slowing down the engine… I have no idea from where that comes.

[illwieckz]

When I reproduce the “PK” bug, all the time is spent in BufferCommandTextInternal()… 96% of CPU usage is in this function, called by `Com_EventLoop'.

[illwieckz]

The “text” processed by BufferCommandTextInternal() is just garbage.

[illwieckz]

I got a crash and the stacktrace doesn't make sense:

#0  Console::Field::AutoComplete(this = 0x555556ef3960 <_ZL11input_field.lto_priv.0>) at /usr/include/c++/13/bits/basic_string.h:1258
#1  CON_Input() at src/engine/sys/con_curses.cpp:508
#2  Com_GetEvent() at src/engine/qcommon/common.cpp:282
#3  Com_EventLoop() at src/engine/qcommon/common.cpp:417
#4  Com_Frame() at src/engine/qcommon/common.cpp:947
#5  Application::ClientApplication::Frame(this = 0x555557690580 <Application::GetApp()::app>) at src/engine/client/ClientApplication.cpp:91
#6  Application::Frame() at src/engine/framework/Application.cpp:87
#7  main(argc = <optimized out>, argv = <optimized out>) at src/engine/framework/System.cpp:1011

It crashed on this assert in the stdlib:

      _GLIBCXX_NODISCARD _GLIBCXX20_CONSTEXPR
      reference
      operator[](size_type __pos)
      {
        // Allow pos == size() both in C++98 mode, as v3 extension,
	// and in C++11 mode.
	__glibcxx_assert(__pos <= size()); ⬅️
        // In pedantic mode be strict in C++98 mode.
	_GLIBCXX_DEBUG_PEDASSERT(__cplusplus >= 201103L || __pos < size());
	return _M_data()[__pos];
      }

And on our side:

			case '\t':
			case KEY_STAB:
				input_field.AutoComplete(); ⬅️
				continue;

[slipher]

I am against adding a third type of pak PAK_BUILTIN. This means we have to deal with a third type of pak anywhere that does pak handling (that includes gamelogic). That would cause us a lot more trouble than it saves.

[illwieckz]

We can do a * filtering like the builtin images.

[illwieckz]

I'm more concerned by this stupid bug btw, alternate ways to implement the flagging of the builtin pak are trivial.

[illwieckz]

Can it be possible that us embedding binary data in the executable breaks some compiler tool that confuses the embedded data with actual code?

[illwieckz]

The bug acts like if the game was spammed with events from the TTY, but that doesn't make sense and the data is so huge and garbage it is more likely processing the engine memory itself after an overflow or so.