Idea dump for truely async resource loading

[floooh]

• currently an entire resource file is loaded into a memory buffer, then a GL resource is created, and the memory buffer content is copied over; the copying happens in the main thread
• it should be possible to create and fill resources (vertex buffers, index buffers, textures) in a separate thread using multiple GL context and shared resource lists (https://www.opengl.org/wiki/OpenGL_and_multithreading)
• it should be possible to incrementally load a file via small fixed-size buffers to avoid memory spikes
• new set of thread-safe resource creation functions called from IO thread, these take a SETUP object, and return an array of pointer/size pairs (e.g. pointer to vertex buffer, index buffer, texture mips), these methods must be THREAD-SAFE, the resource slot has already been reserved on the main thread in the Gfx::Resource().Load() method:
  • Array<PointerAndSize> BeginSetupThreaded(Id id, const SETUP& setup);
  • void EndSetupThreaded(Id id);
• between BeginSetupAsync() and EndSetupAsync() a loader thread will fill the returned memory areas and when done will call EndSetupAsync(), during this, the resource slot will be locked with a separate flag. Once the locking flag is cleared, the resource pool slot will switch from Pending to Valid (on the main thread), from then on it is safe to use the resource
• question: how to implement destroying the resource while an async load is in progress? => mark as deleted, let resource loading finish, then destroy on main thread(?)
• question: one separate loader object per resource? loader probably needs to keep and update tracking state during the loading
• loader needs a generic OnDataAvailable(void* ptr, int32 size) method which can be called with any size by an IO thread (or on some platforms on the main thread), usually loader needs to read enough data until a file header is complete, then call BeginSetupAsync() to create the GL resource(s), and write the rest of the data into the returned pointers.
• question: how to communicate loading errors and 'finished' to loader?

[floooh]

No threaded loading with multiple GL contexts, instead (later) for desktop GL (or all implementations which have mapping: (pre-)allocate GL resource(s), map-buffer in main thread, fill in worker thread, unmap in main thread, render. Probably need to separate between loader thread(s) and resource creation thread.

[floooh]

How a loader should actually work:

• complete resource loading and creation in several steps:
• on main thread: allocate resource id, set resource state to pending, create and fire ioRequest, watch for ioRequest to be handled
• on IO thread: have an optional method to decode loaded data (e.g. to parse a file format into a canonical format), but don't create any GL resources in loader thread.
• back on main thread (after ioReq has been marked as handled), either create resource directly, or delegate to a resource creation thread
• Loader sets resource state to Valid or Failed, and is done
• GfxResourceContainer should put all Loaders into a queue and trigger them as long as loader is not done.

[floooh]

Perfect resource streaming only possible on GL4.x+ (http://www.slideshare.net/CassEveritt/approaching-zero-driver-overhead). So need to implement at least 2 paths for GL2.1+exts, and "full AZDO" with GL4.x.

Also: pre-allocate and map one big vertex buffer, and let dlmalloc (or other existing allocator) do the allocation in this buffer? Or allow true streaming only with restrictions? (like knowing the required resource buffer sizes beforehand?)

[gustav3d]

or screw it for the the old GL2.1 path and implement it only on modern hardware ?