Editor data format

[Moneyl]

At the moment Nanoforge interacts with RFG data formats directly. The problem is that those formats are optimized for fast loading, whereas Nanoforge requires ease of editing. I believe this is making the code overly complicated. The plan is to design a data format around the requirements of Nanoforge. RFG formats will be converted to this format before use. This has several benefits:

• Code related to RFG formats would be isolated to RfgTools++ and import/export functions. The design choices of RFGs data format don't seep into the rest of the codebase.
• Reflection. Editor format will have property names and types built in. Can auto generate UIs and serialization.
• More uniform code. All tools will work with the editor format instead of each of them being unique.
• Built in edit tracking for undo/redo. Also won't need to implement this separately for each RFG format.
• Easy integration with scripting languages.

Format design

The design for the editor format isn't complete, but here's what's known so far. The format consists of objects with properties. Properties are primitive data like integers, floats, strings, object references, object lists, or buffers. Objects are owned by a registry which provides handles to access them with. Each object has a unique ID (64 bit uint). Properties can be added/removed at runtime.

The editor format can be serialized in any form. It'll likely use a text format by default to be version control friendly. One exception is buffer properties. These are binary blobs, like vertex buffers for a mesh, or pixels for a texture. They'll be stored in separate binary files and loaded on demand.

Roadmap

The map editor is going to be converted to this format as an initial test case. It should make the implementation of the map editor simpler and give some insight into how the design works. That way any major adjustments are made prior to converting the entire codebase to this design. These changes are being made on the master branch.

[Moneyl]

Just finished converting the map viewer to use the editor data format. Now the first time a map is opened it has to be imported. This converts the zones and terrain to registry objects, with meshes and textures stored as binary blobs in buffer files. Then subsequent loads use the registry data. This had a side effect of making loading much quicker after the initial import. Loading the main map took ~40s on my system on v0.19.x. Now it takes 1.3s with the initial import taking ~2m40s.

Editor objects are serialized to text files and buffers are saved as binary files. A text format is used so has nice git diffs. Its using a custom text format since the xml libraries I tried either loaded or saved the files too slowly. The main map is ~750MB after import. 90MB of text and 660MB of buffers (meshes and textures). Eventually I'll add support for re-importing the meshes and textures from packfiles on demand so they can be excluded from git repos.

Next up I'm going to work on the map editor to test the editor format further. Some things to note about the editor format so far:

• Multi-threading could be handled better. Right now each object has a std::mutex. This uses a lot of memory (80000+ objects in the main map).
• Properties are stored in std::variant<>, so even a bool uses 48 bytes. Could eek out some size savings by storing properties in a contiguous buffer. I think this would also allow mutex free threading using std::atomic_compare_exchange_weak_explicit, but I need to look into it further.
• Edit tracking & undo/redo aren't supported yet.

[Moneyl]

Now that I've gotten a chance to properly test this design with live code, I see some issues with it:

1. Since properties are accessed via strings, typos can result in a runtime error.
2. Using the wrong type in a property accessor results in a runtime error.
3. The performance and memory usage are poor.

There are ways to improve these problems. Some of the performance improvements were mentioned in the last message. However, I think I'd prefer a different set of trade-offs. I'm taking a different approach in the Beeflang rewrite of Nanoforge:

• Editor objects are plain class instances and properties are fields on those classes. This is much more convenient than using string accessors, and you get compiler checks that prevent issues 1 & 2. Beef also has reflection so the system doesn't need to track all kinds of type information like it did with C++.
• Undo/redo will be implemented from the start since that'll impact how a lot of code is written. The current plan for change tracking is to use a transaction system similar to what's mentioned in this GDC talk Creating a Tools Pipeline for Horizon: Zero Dawn.
• This design won't be "magically thread-safe" like the original theoretically was. It'll be the onus of the programmer to ensure two threads aren't modifying an object simultaneously.

Overall this design is trading easy thread-safety and flexibility for ease of use, performance, and compiler safety checks. I think it'll result in less bugs, be easier and quicker to program with, and have less memory and CPU usage by default. I'm in the process of writing it now. Once I get the map editor working again I should have a good idea of how effective this design was so I'll report back then.

[Moneyl]

Closing this since the base implementation is in the NF rewrite. You define plain classes which inherit EditorObject and get serialized by ProjectDB using the bon library. Undo/redo is incomplete and won't be completed until after v1.0.0 is released.