winit
winit copied to clipboard
rust-windowing
Implement VSync source
This is a common concept in e.g. browsers to have non blocking frame scheduling at the display refresh rate.
It seems like it's possible on all desktop platforms and the web. For the rest we may indicate that vsync source is not supported.
The way to integrate it would be to add WindowEvent::Frame which must be send when the new frame is ready to be drawn. That will come from the compositor directly and the users must request those event via Window::request_frame() -> Result<()>, where the result will indicate that the event can't be scheduled due to reasons.
The Api that should be used for that on platform are the following.
X11 - Xpresent
macOS - DisplayLink
Wayland - frame callbacks
Windows - DWM get composition timings + a timer.
Web - request animation frame.
The Api is essential for Wayland clients, since they must not rely on vsync from the graphics Api.
Sounds good to me, though it would be worth considering how failures of request_frame should be handled (e.g. requesting a wake in 16ms?).
should be handled (e.g. requesting a wake in 16ms?).
I think it's up to application what to do here. Like it clearly knows how that it failed before hand, so can just fallback to something else. I don't want to add timers to every backend, just to provide some hand holding. At least for now. In general this can only fail on X11 if you don't have libXpresent installed, so it's like you should really try to fail...
Based on discussing this a bit on IRC I think it could be good to highlight one clear sub-problem that came up which is that it could be really helpful to have just one place where applications are allowed to render.
e.g. in the case of Wayland, if it was guaranteed that apps only rendered during RedrawRequested event callbacks then the backend could potentially register frame callbacks there to get notifications from the compositor for throttling the application.
(so potentially frame throttling might be possible on Wayland with no new API if it were documented that rendering were limited to RedrawRequested events. Similarly this should be possible for X11).
Thinking more generally about high level APIs related to rendering synchronization may open a bigger can of worms, which could be good to try and break down.
One initial concern I have about a request_frame() API that seems inspired by requestAnimationFrame in browsers is that it looks kind of back-to-front compared to typical presentation APIs that would throttle / synchronize you based on a previously submitted frame.
At a lower level then synchronization with the vblank will either be handled somewhat transparently by apis like eglSwapBuffers that can be configured to block while they wait the next vblank to release a new backbuffer or may be done more explicitly with a fence object api of some form that lets the application poll (or get an event) for when work on the gpu has completed. These mechanisms only really apply to non-composited applications though.
In composited environments the norm is to get asynchronous feedback after you give the compositor a frame that may tell you when the frame became visible for the user or at least when the compositor processed the frame. For basic throttling of the application then the exact semantics don't necessarily matter too much, just so long as it's a 1:1 ratio for vblank.
In both cases though the synchronization (or I'd say throttling for a composited environment) comes after you have rendered something and submitted it to be presented.
In general though it's probably good to keep in mind the differences between composited / non-composited applications here (presentation being throttled by a compositor vs synchronized with real hardware) which will mainly affect how the app gets its feedback. In the case of eglSwapBuffers the feedback may just be implicit (the app process may be blocked until a new back buffer is ready)
Off the top of my head I can think of three main points of interest and kinds of synchronization that applications are likely to be interested in for different use cases:
- The compositor has finished processing the frame that included your app-rendered frame (this is what wayland frame callbacks are generally about)
- The compositor has rendered its own frame that includes your app-rendered frame and that is now visible to the user on a screen (E.g. some X11 compositors can give info about this, and I think there was probably also wayland protocol for this too)
- Now is a "good" time to start rendering to have the minimum time between rendering and having your frame be visible to a user. (As in scheduling rendering to minimize latency between what's rendered and seen - important for VR, also called frame pacing, E.g. see https://developer.android.com/games/sdk/frame-pacing)
The above request_frame() API seems like might be more suited to trying to support frame pacing, which is more about trying to use heuristics to predict the best time for an application to start rendering. I'd generally say this is a much harder, and also more subjective, nuanced problem space than the problem of throttling rendering to the VSync, which is what the title of this issue references.
Libraries like Swappy on Android might help with being able to support frame pacing via Winit but it's the kind of thing that needs to use some amount of guess work and heuristics and I think it could be quite challenging to stabilize for lots of general purpose use cases across lots of window systems.
Looking just at Swappy for Android you can see here https://developer.android.com/games/sdk/frame-pacing/opengl/add-functions how the library also depends on graphics driver extensions which will probably complicate the relationship between Winit and APIs like Wgpu.
If there would be real interest in re-considering quite how Winit drives its rendering, such as to help with synchronization/throttling/pacing, it could also be a good opportunity to re-consider what would be ideal for iOS and OSX which I generally recall drive their rendering via delegate callbacks where apps are really expected to do, and finish their rendering synchronously as part of those callbacks whenever they happen (which doesn't fit super well with the winit event model as far as I recall)... just found the issue I was thinking of: https://github.com/rust-windowing/winit/issues/2010
On Android we also want to be very clear about where rendering happens considering that apps must not render if they don't have an associated surface, so I'd definitely be interested in seeing Winit clearly limit/define where applications are allowed to render.
I think what I really want here is a frame throttling source, so I can draw at the vsync rate without blocking and with very low chances of missing vblanks.
I've started the work here https://github.com/rust-windowing/winit/pull/2535 for X11 for now, since I can't add Wayland ergonomically here.
Assuming that we're dealing with composited applications (i.e. ignoring the possibility of writing compositors using Winit for now) then yeah it seems good to prioritize having a way the throttle client rendering, which may not happen automatically with APIs like egl/glxSwapBuffers.
-
For X11 (as mentioned under #2535) my instinct would be to look at using the
_NET_WM_FRAME_DRAWNprotocol. -
For Wayland I'd expect to use frame callbacks.
-
I'm much less sure about Windows but there is a
DwmFlush()API that can sync (blocking) with the compositor (which seems excessive from the main thread). There's alsoDwmGetCompositionTiming. I have a feeling that on windows you can callDwmFlush()from a thread, and so combined withDwmGetCompositionTimingit may be possible to create a mechanism for waking the event loop in sync with the compositor. -
On macOS / iOS it looks like the recommended way to throttle rendering is via a
CADisplayLink /CVDisplayLinkcallback which abstracts away from synchronizing with any compositor, but should approximately throttle to the vsync. -
On Android, without going down the route of using Swappy for frame pacing then it looks like we can instead get a vsync callback from the surface flinger via a "choreographer", which is also available via the NDK: https://developer.android.com/ndk/reference/group/choreographer
-
For web I suppose the closest thing would be to use
requestAnimationFrame
ah, just realized that the original issue description listed most of the above, sorry
For X11, Wayland and maybe Windows then it seems like there's more capability to track the progress of specific frames (or on Android with the extensions that Swappy uses) where you can get detailed stats/metrics relating to a specific frame.
For macOS/iOS/Android (choreographer)/web you essentially just get a higher-level callback mechanism that will generally throttle your rendering but without any detailed per-frame feedback.
I think it would be nice if Winit could support both of these general models:
- a way to get per-frame throttling and timing feedback if supported by the window system or
- a general frame pacing callback/event
but I guess that the easiest abstraction to support initially in a portable way would just be (2) to have a frame pacing callback/event across platforms.
Based on some of the earlier discussion it might also be good to consider making such a callback/event be the only place that applications are allowed to render (good for Wayland).
It might also be possible to just define that the existing RedrawRequested event would be that event.
It's maybe worth highlighting here that the #2535 discussion has been a good reminder of how much of a mine field this is for X11. This capability was something that was worked on mainly in support of Intel drivers and Gnome 3 way back around 2010-2013 but even to this day there's not really any one standard way to support throttling of x11 clients to vblank/the compositor and it's also a fairly disproportionate amount of work to try and cover all the unique cases, including:
- windowed client, x11 compositor, open source drivers
- full screen client, x11 compositor (potentially unredirected), open source drivers
- windowed client, x11 compositor, nvidia drivers
- full screen client, x11 compositor (potentially unredirected), nvidia drivers
- windowed client, xwayland assuming open source drivers (since wayland not generally supported on nvidia)
- full screen client, xwayland (potentially unredirected), assuming O/S drivers
- windowed client, non-composited, open source drivers
- full screen client, non-composited, open source drivers
- windowed client, non-composited, nvidia drivers
- full screen client, non-composited, nvidia drivers
It's pretty much no joke that each of those cases need special consideration and a mixture of different technical solutions that aren't very consistent with each other. Sometimes the ideal solution depends on glx/egl extensions but with inconsistency between glx + egl support (both out of scope of Winit currently). The situation with Vulkan is likely even worse in some cases because since Vulkan has emerged there has been comparatively little development on x11 (the main case of handling full screen games would be the priority)
From what I can say, the only option for now would be not support X11 at all. And add support for _NET_WM_FRAME_DRAWN will be adopted by picom/kwin for example. Xpresent shown that it's not really good solution and works only with Xwayland(which is good for games though, but there's no way to detect that we're under XWayland).
So I guess we may just go without X11 for now, it's not really a big deal given that Wayland is where all the cool work is happening and X11 support is more about providing better experience for software that no-one cares about maintaining anymore.
Yeah, I'm mostly all for forgetting about X11 at this point ;-p
Unfortunately, in reality Wayland still isn't supported on Nvidia and so a large chunk of Linux desktop users are likely to still be stuck using X11 which does spoil the picture here somewhat.
Maybe one way of starting to think of things could be that classic X11 likely implies that you're running on Nvidia and everyone with open source drivers is probably (or should at least have the option of) running a Wayland compositor / xwayland. Though it's not quite that simple - I have a hybrid Razer laptop with integrated (Intel) graphics and a discrete Nvidia GPU and that's also awkward to run Wayland on.
but there's no way to detect that we're under XWayland
I imagine if this would be helpful we can figure out a way of detecting this with a reasonable amount of certainty (though regarding xpresent I think we'd be playing with fire to be trying to use the protocol directly outside of the EGL/GLX driver)
Well, the recent nvidia stuff does work on Wayland from what I see and some folks run sway on nvidia binary drivers, so it's changing from what I can say.
Yeah actually I just had my mind blown while looking at the mutter source code re: Nvidia support.
I was just looking through some of the mutter code out of curiosity to see if I could find a standard enough atom that would e.g. be set on the "root" window to identify xwayland and I just saw that they have been adding EGLStream support to mutter!
EGLStreams is the extension that Nvidia have always pushed for using as the basis for supporting Wayland which has historically been rejected by the open source xorg/freedesktop community (since it means having a special case for lots of things just to support a single vendor via a vendor-specific EGL extension)
It looks like that was started around december 2021 and there's still more recent work that's been happening for supporting EGLStreams so maybe we're finally going to get to the point where we really can say good riddance to classic X11!
That's made my day!
Considering that I bootstrapped the Wayland support in Mutter I was always a bit disappointed that for years afterwards the maintainers took the principled stance against supporting Nvidia's EGLStream extension vs being (imho) more pragmatic about the bigger benefits that could come from being able to move the community forwards to Wayland at least (and just argue about finding a common technical solution to replace EGLStreams + gbm later)
There are actually reasonable pros and cons arguments for EGLStreams vs gbm (which is also not a perfect solution) so I think we could have been more compromising early on here.
If you look at comsic-comp (The pop os compositor) it should support EGL streams, since smithay supports EGL streams, but EGL streams are not that required nowadays given that NVIDIA has drm/gbm support now. You just need a recent driver and luck that it'll work with your setup.
Some of their work to support drm/kms/gbm looked like it was very limited, and only usable for some very narrow use cases but maybe it's improved. I wouldn't be surprised it's it's still much better to just use EGLStreams for Nvidia for anything but bare bones support. At least in mutter it looks like they are actively working on EGLStreams support so I guess it's still the more practical way of support NVidia vs using their gbm support.
I think for now we'll go with just a throttling hint given that frame timings are extra on top of it most of the time and some clients might not provide it. At least this will solve the problem for the web and Wayland backends, and the rest will use fallbacks.
Drawing on such throttling hint is not what applications always want, since they could have a state where nothing should be drawn. The only caveat with such API is that you should do the drawing after requesting, this assumption will be on our users, given that winit doesn't draw or control the drawing itself.
I don't know much about the other backends besides Web, but maybe we could draw the line at throttling hints. Wgpu is planning to implement an API to get precise timings for frame pacing: https://github.com/gfx-rs/wgpu/issues/2869.
So if Winit offers throttling hints and Wgpu offers timings for frame pacing, that seems like a good separation of responsibilities to me.
So if Winit offers throttling hints and Wgpu offers timings for frame pacing, that seems like a good separation of responsibilities to me.
winit could get this as well(from display server, it's available on at least all desktop platforms), though it's true that graphics drivers also provide such information and some provide extra methods to manipulate it.
Throttling could be added because it's not controversial at least.
winit could get this as well(from display server, it's available on at least all desktop platforms)
Ah, I wasn't aware, sounds good to me then.
winit could get this as well(from display server, it's available on at least all desktop platforms), though it's true that graphics drivers also provide such information and some provide extra methods to manipulate it.
I think it's unlikely that there's much redundancy for getting the same information on a specific OS.
One of the challenges here is that different OSs expose similar things in rather different ways - this makes it challenging to figure out a neat way of exposing some of these details in a portable way, especially if Winit tries to avoid graphics driver details, and Wgpu tries to avoid window system details.
That said though I agree that helping with throttling would be a good starting point and I think there's relatively low hanging fruit where Winit can help here.
With the pump events branch I made some progress towards making RedrawRequested the event that applications can use to drive their rendering which should naturally throttle rendering for Web, Wayland, Windows, macOS and iOS.
I think the RedrawRequested event is already a reasonable way to expose frame scheduling/throttling for Winit.
- On Web RedrawRequested can be driven by requestAnimationFrame
- On Wayland RedrawRequested can be driven by frame callbacks
- On macOS RedrawRequested can be driven by drawRect
- On Windows RedrawRequested can be driven by WM_PAINT
We can then clearly document that applications should drive their rendering via RedrawRequested events exclusively.
That seems like it would be a good first goal for Winit to at least drive redraws based on the throttled events that the window systems provide.
There would certainly be ways to improve the scheduling on different OSs but as a baseline that seems like it could be better than the current situation.
We also had some good discussion about some of this stuff here: https://github.com/rust-windowing/winit/issues/2640
I think the RedrawRequested event is already a reasonable way to expose frame scheduling/throttling for Winit.
I don't think it's a good idea tbh. We need an event to tell application to redraw, which on web it's not the case, so right now we don't even send this event, because OS never asks you to draw. However on macOS/Wayland you could have this event for sure when the OS simply forces you to redraw, regardless of the frame scheduling. This event is not only about contiguous drawing in the end of the day.
The application may even not use the frame throttling hints and rely on the graphics driver, so it's not that great either here.
However what we can do instead is track when the application uses frame callbacks and throttle other UI events, like squash resizes under the hood and so on, so for example it'll mitigate EGL resizing buffer latching on Wayland, because you'll resize only once per frame callback.
We can then clearly document that applications should drive their rendering via RedrawRequested events exclusively.
My main issue with that is that sometimes you should force the client to submit the buffer regardless of what throttling you're using, it's true that we basically have a way to request a redraw and a way to throttle frames, and those events sound really similar.
I'd give an example on how RedrawRequested is being used in the wild by alacritty. This all happens inside the run.
match event {
WindowEvent::RedrawRequested => dirty = true;
WindowEvent::FrameThrottled => has_frame = true;
}
if !has_frame {
return;
}
if dirty {
dirty = false;
do_gl_stuff();
has_frame = false;
request_frame().or_schedule_timer();
swap_buffers();
}
It's true that we could throttle the redrawing via the RedrawRequested by naturally throttling it, however the issue is how should we limit fps on macOS/Windows, what to do with the platforms we can't? The proposed API in https://github.com/rust-windowing/winit/pull/2883 will handle such cases by telling the users that they should do it on their own, with RedrawRequested we can't unless we redesign the entire event.
Winit shouldn't need to have separate events to distinguish when a redraw is technically required due to window-system specific details (such as Expose events on non-composited x11) or e.g. resizing of windows, that would be an over-complicated application programming model imho.
It should be ok that there's one event apps can drive redraws from, which 99% of the time arrives because the app has requested it or may come because the OS requires it.
One thing that's probably missing from Winit's design currently though is that request_redraw should probably accept damage rectangles/regions so that RedrawRequested events can echo back the damaged regions.
That would be more consistent e.g. with the design of drawRect that can tell you a region to redraw, as well as X11 Expose events as well as WM_PAINT update regions.
Being able to forward a region that needs to be redrawn lets you just have one mechanism because the OS can simply emit events with a full-window region if it requires you to redraw everything, or else the events can provide a union of the regions that where damaged via request_redraw.
It's true that we could throttle the redrawing via the RedrawRequested by naturally throttling it, however the issue is how should we limit fps on macOS/Windows
We can use drawRect on macOS as suggested which should throttle? macOS is specifically one of the platforms where we really want to remove the emission of RedrawRequested after MainEventsCleared and should pretty much end up implementing the throttling your after out of the box.
and WM_PAINT on windows which should also throttle
I'm pretty sure we're talking past each others.
If I as an application request a throttling hint, but I don't want to draw on receiving it, how I should handle that with your idea to merging the events? Why should we care about damage regions in the first place now, given that they are emitted by, for example egl/vulkan/wgpu calls under the hood?
The main point is that my application want to request frames for contiguous animations, and in some cases my application doesn't want to do any drawing in reaction to FrameThrottled but simply wait until its scene changes and do an instant drawing, if we make change the RedrawRequested to a point of DrawNow then ok, sure, but it's not really clear that we really should do the draw? I think on macOS we must draw in drawRect, because the OS calls back to you. And how should we deal with user requested redraws? We can't call drawRect ourselves, so we must call it from other place like it is now.
From the drawRect docs
This method is called when a view is first displayed or when an event occurs that invalidates a visible part of the view. You should never call this method directly yourself. To invalidate part of your view, and thus cause that portion to be redrawn, call the setNeedsDisplay or setNeedsDisplayInRect: method instead.
So it seems like drawRect calling is done by the os, which is not what the event I'm proposing does or could possibly do, it's exactly what RedrawRequested is for, and it means that the sync operation should be done.
I'm pretty sure we're talking past each others.
yeah, I guess maybe. I can try and be more specific with how I'm imagining the implementation would work in my mind to see if we can figure how we're talking past each other.
If I as an application request a throttling hint, but I don't want to draw on receiving it,
tbh this isn't a concept I would expose to applications - I think RedrawRequested events can be implicitly throttled because most window systems have mechanisms (such as drawRect, WM_PAINT, frame callbacks, requestAnimationFrame) we can utilize for this without requiring applications to request anything.
The main point is that my application want to request frames for contiguous animations
yeah, and in this case your app would be calling request_redraw while animating which would result in a future RedrawRequested event.
If you aren't animating and nothing has changed you won't request a redraw and so you wont get RedrawRequested events. (except for window-system specific special cases like windows resizes which you need to handle even if you aren't animating)
On macOS after you request a redraw, then the backend will request a callback from drawRect and then the next time that drawRect is called then the application will get a RedrawRequested event which the app can then render within, properly synchronized in the way macOS/iOS wants all rendering to be synchronized.
On Windows after you request a redraw then the backend will request a WM_PAINT event and the next time that it receives a WM_PAINT (throttled by the OS) it will emit a RedrawRequested event for the app.
On Wayland after you request a redraw Winit can dispatch that e.g. after MainEventsCleared (wayland doesn't care when the client renders) but you will never emit RedrawRequested more frequently than one per frame callback, so sometimes you may throttle dispatching the RedrawRequested until the next frame callback arrives.
Similarly on Web after you request a redraw Winit can dispatch a RedrawRequested event asap but never more frequently than once per requestAnimationFrame callback.
tbh this isn't a concept I would expose to applications - I think RedrawRequested events can be implicitly throttled because most window systems have mechanisms (such as drawRect, WM_PAINT, frame callbacks, requestAnimationFrame) we can utilize for this without requiring applications to request anything.
So in such case when the application doesn't want to draw anything it simply ignores the event?
I mean, if we pass a state on how sent the RedrawRequested back like system or user and if we warn when user request went None we could probably do that, we just can't keep the event as is and change its semantics, and we'd need some indication when there's a must to redraw (it'll send a System in it, on Wayland it'll soon be the case, when the protocol to get new buffers will be present, and on macOS it'll be the case on startup and some internal operations, like Resize).
tbh this isn't a concept I would expose to applications - I think RedrawRequested events can be implicitly throttled because most window systems have mechanisms (such as drawRect, WM_PAINT, frame callbacks, requestAnimationFrame) we can utilize for this without requiring applications to request anything.
So in such case when the application doesn't want to draw anything it simply ignores the event?
No you shouldn't be ignoring the event because it's either come because the window system requires it (e.g. window resize) or it's come because your app requested it.
If you don't want to draw anything then you don't need to request to redraw and so you will only get RedrawRequested events for event that you must redraw for.
I mean, I know how it sort of works, the issue is that we try to revoke semantics of the established event to completely different semantics. I'm fine with such event if we can somehow indicate via it that redraw must be performed regardless of user decisions. So if we settle with RedrawRequested(User/System) meaning that System forces redraw and User means, that it's your request and you may not draw on it, then it's probably will work just fine. It'll be squashed under the hood with system taking preference over used and being throttled the way it expects to be throttled.
I'm just afraid that we'd need yet another event telling clients that they must resubmit their buffers, which is a real world use case on the present systems.
If you don't want to draw anything then you don't need to request to redraw and so you will only get RedrawRequested events for event that you must redraw for.
But that's an issue then? On Wayland you must draw after asking for a frame and the frame event is more of a hint, not a must. You may not know before hand that you don't want to draw, because by the time you wait for the frame back, the state might change and you may want to draw.
We have a bit of a zig zag conversation going atm, but just to note that I commented here https://github.com/rust-windowing/winit/pull/2883#issuecomment-1595863774 that I'm not suggesting that the user/system distinction needs to be exposed to apps.