Archive/Game Package System

[equalent]

Hello again. It would be very good if a resource packing system was built into bs::f (implemented as a virtual filesystem), such as the resource packing system in Urho3D. It may also support compression, for example, LZ4 (very popular for game development). Is this system already in development or is planned for development?

Best, Andrey

[BearishSun]

What would be the advantages and use cases of such a system compared to keeping the files separate?

[equalent]

Well, there will be many advantages:

• Size optimization (most important) - no extra file header information and compression
• Performance optimization - bs::f can split all resources in different groups and minimize disk seeks during IO operations
• Deployment simplicity - it's easier to manage several files instead of several thousand (for huge projects)
• Some kind of data obfuscation

[equalent]

And it's simply more elegant to have this instead of this For example the implementation in Source Engine by Valve is very simple: we have a "*_dir.vpk" file which contains all virtual FS info (directories, etc.) and other "*.vpk" files which simply store the binary data.

[BearishSun]

Note that compression is already supported in the current system.

Eventually I would like to add some form of resource packaging, but priority wise it's going to be after a lot of the more major features that are yet to be added. I'd gladly accept a PR on the topic if you (or anyone) wishes to add such a system.

While I see a benefit with this system regarding random resource loading in terms of reducing individual file overhead (which will be a problem for very large projects as you said), the main benefit for me would perhaps be to group resources by relevance (e.g. scene/type) load/unload them in bulk (a package acting more like one big resource instead of a VFS).

[equalent]

So these packages should act like Unity's AssetBundle? No automatic resource loading based on the path? For example:

HPackage package = Package::create();
package->addResource(texture, "textures/characters/zombie_albedo.asset");
gResources().save(package, "package.bpk");

[BearishSun]

Similar to asset bundles, but also with the ability to load individual assets from the package without having to load the whole thing (I don't think Unity asset bundles support that but I might be wrong). This might be controlled by a flag on package creation (or at load time if possible).

I think we probably want automatic resource loading as well. Perhaps we let the package be registered with the Resources system similar how ResourceManifest (which holds UUID -> disk path mappings) is registered right now - and then the system auto-resolves the assets.

HPackage package = ... load the package (header/meta-data only, no resource loading at this point) ...;
gResources().addPackage(package);

[equalent]

Yeah, I think that's better than loading full packages. But how can we load separate resources if we don't use VFS? Maybe...

HPackage package = ... load the package (header/meta-data only, no resource loading at this point) ...;
gResources().addPackage(package);
HTexture texture = gResources().loadPackageResource<Texture>(package, "abc.asset");

That's absurdly :) Without VFS we can only implement separate loading.

[BearishSun]

Loading an asset directly:

HPackage package = Package::load("myPackage.bpk"); // Loads meta-data only
package->load("abc.asset");

Registering an asset for automatic resolving:

HPackage package = Package::load("myPackage.bpk"); // Loads meta-data only
gResources().addPackage(package);

Loading the whole package:

HPackage package = Package::loadAll("myAnotherPackage.bpk");

Or something along those lines...

[ShaydeNZ]

Hi there.

I wanted to add that there's other use cases for being able to load from other than files; for instance, loading assets from a web-server, or from the game server for a multiplayer game.

The current system appears to expect resources to be loaded from file, which seems a bit inflexible. More-so given the FileSystem class already provides the ability to stream from sources other than from file (e.g. memory). The resource loaders would ideally come with versions that take a DataStream, so developers can roll their own asset bundle providers, or remote asset providers, as required. This is pretty standard for most engines I've worked with, and I must admit it was surprising to find bsf doesn't do this.

I suppose it is possible to just extract, or download, assets into a temporary file then do a resource load from that, but it's not very elegant.

[BearishSun]

It's not as simple as just passing along a DataStream as that would reserve an entire thread/core for just downloading a resource if passed to the resource system as currently implemented. The system would need to be modified so it can relinquish its thread if waiting for very long I/O commands, such as networked resources.

That said I am aware of the use case and plan on adding support for it eventually, but more critical features have priority. If you are interested in adding this functionality yourself it would be welcome.