Asynchronous file I/O

[SDLBugzilla]

This bug report was migrated from our old Bugzilla tracker.

Reported in version: HG 2.1 Reported for operating system, platform: All, All

Comments on the original bug report:

On 2014-02-24 01:55:02 +0000, Nathaniel Fries wrote:

Currently, file RWops are blocking and synchronous (just like what's provided by the C FILE* and C++ fstream). This is all fine in good for non-interactive programs and initialization-time loading. But most applications of SDL are interactive, and many are games and media players, which need to read from the harddisk while remaining interactive.

Because file RWops are blocking (take control away from the main thread), reading too much data from the hard disk on the main thread makes programs non-interactive. This leads to messy solutions like creating another thread just to load data ( "Main thread renders loading animation while background thread uploads whole level along with textures. In fact I did notice that this takes slightly longer than doing everything in main but user experience is much better with main thread still operational, showing anims and gameplay tips. ": http://forums.libsdl.org/viewtopic.php?t=9897 ).

So, I request the addition of asynchronous I/O features to SDL.

POSIX non-blocking I/O is simple: pass the flag O_NONBLOCK to open(), and all calls to read/write will return without waiting for the disk (the return value will be the number of bytes written). Asynchronous file I/O was added to POSIX in 2001. Most versions of the interface do not work on sockets. See: http://pubs.opengroup.org/onlinepubs/009604599/basedefs/aio.h.html Non-blocking I/O can be multiplexed using select() in order to reduce the number of read()/write() syscalls: http://pubs.opengroup.org/onlinepubs/7908799/xsh/select.html With non-blocking I/O, SDL will have to maintain the total number of bytes written in order to determine when an I/O operation has completed.

Windows does not support synchronous (in order) non-blocking I/O. Instead it only supports asynchronous (out-of-order, or in Windows terms, overlapped) I/O. This is implemented by passing FILE_FLAG_OVERLAPPED to CreateFile and maintaining an OVERLAPPED structure for each I/O operation. See: http://msdn.microsoft.com/en-us/library/windows/desktop/aa365683%28v=vs.85%29.aspx There are several ways to get indications of I/O completions using asynchronous I/O. One is to associate a kernel event with the OVERLAPPED structure passed to ReadFile/WriteFile. Another is to call GetOverlappedResult. But because an application that has real use of asynchronous I/O is probably performing operations on multiple files, it would be better to reduce the number of syscalls. This means using Read/WriteFileEx and entering an alertable wait state in the main loop (a new API function, SDL_PumpIO, for other threads; and probably inside SDL_PumpEvents) to execute I/O completion handlers. See: http://msdn.microsoft.com/en-us/library/windows/desktop/aa363772%28v=vs.85%29.aspx

It is entirely possible to implement this using the existing SDL_RWops interface. But an asynchronous SDL_RWop would break old code using SDL_RWops by its nature.

Instead, I recommend an additional API just for asynchronous file ops. While this could certainly be done as a third party library, I feel that it would be a better fit for the main library, using the SDL event queue to report completed/failed async ops with a new event.

On 2017-08-14 20:42:19 +0000, Sam Lantinga wrote:

This is a good idea for SDL 2.1. Do you have a proposed first pass implementation?