tokio
tokio copied to clipboard
tokio-rs
Provide boottime timers
Is your feature request related to a problem? Please describe.
On linux, tokio currently delegates to std::sync::Condvar::wait_timeout and epoll_wait. These functions use CLOCK_MONOTONIC. Therefore, time spent in system sleep does not count towards tokio timers.
Sometimes you want to schedule an operation periodically. For example, poll an external resource once an hour. CLOCK_MONOTONIC is not suitable for this usecase. After the system wakes up from sleep, you most likely want to poll the external resource immediately if a wall clock hour has elapsed.
Describe the solution you'd like
Provide some way to create CLOCK_BOOTTIME timers.
Describe alternatives you've considered
This can be implemented as an external library. But tokio already contains the infrastructure to work with timers.
Additional context
I assume timeouts are also affected by sleep not counting towards them.
On Windows, std::sync::Condvar::wait_timeout behaves the same way. However, due to https://github.com/rust-lang/rust/issues/79462, any spurious wakeup will cause the timer to expire.
I'm not too familiar with the timing facilities. Which OS waiting functionality should we use to hook into these timers?
I believe among platforms only Posix/Linux allows to configure which clock to use
- On posix, the clock used by condition variables can be set. However, linux only support the monotonic and the realtime clock.
- On linux, epoll_wait only supports the monotonic clock.
- On linux, iouring only supports the monotonic clock.
- On linux, timerfds support the boottime clock.
I've checked the epoll_wait implementation and I haven't seen any reason for this restriction except that nobody has implemented support for other clocks yet. The commit that added boottime support to timerfds was trivial and epoll_wait seems to be using the same infrastructure for timers: https://github.com/torvalds/linux/commit/4a2378a943f09
The only way to use boottime timers with all currently supported linux versions seems to be via timerfds. This integrates nicely with the existing tokio infrastructure via AsyncFd.
I haven't looked at other operating systems.
I haven't looked at other operating systems.
Since tokio is cross-platform, it would need consistent behavior across all platforms. Which seems to be lacking at least as far as I know
I haven't looked at other operating systems.
On OpenBSD and macOS, the monotonic clocks are boottime clocks
FreeBSD does not seem to have a boottime clock.
Windows does not have any direct support for boottime sleep. However, you can use RegisterSuspendResumeNotification to get notified upon suspend/resume so that you can adjust the existing timers accordingly.
I haven't looked at other operating systems.
Since tokio is cross-platform, it would need consistent behavior across all platforms. Which seems to be lacking at least as far as I know
The current behavior is not consistent because
- sleep on linux uses CLOCK_MONOTONIC which counts uptime
- sleep on macos uses CLOCK_MONOTONIC which counts boottime
- sleep on windows uses uptime but time is measured using bootime. A sleep on windows can end any time between the boottime expiration and the uptime expiration. It's not reliable.
This has come up again with std::time::Instant.
In our use case, a networking daemon with long-lived connections (~3 minutes) is losing connections without realizing it on Android because the device has been suspended. This results in the service being unresponsive for up to three minutes after unlock if the suspend came at a perfectly bad time, and having hit more than a minute in the field when unlucky. We've tried to work around it, but without timeout support for CLOCK_BOOTTIME or similar, our workaround is incomplete.
Even if std changes to use CLOCK_BOOTTIME for Instant (which is not yet a given), tokio would need to use a timerfd inside epoll as suggested by @mahkoh in order to provide a version of tokio::time::timeout that ticks during suspend.
While I am of the opinion that tokio::time should be converted to use CLOCK_BOOTTIME (including the timerfd usage), I could also see creating a parallel tokio::boot_time module or similar in order to expose this functionality without a change in behavior if we have some reason to believe that someone might be depending on CLOCK_MONOTONIC behavior.
tl;dr: In order for tokio to be usable for on-device settings, we need timers which tick during suspend - these devices suspend frequently and for long periods of time, and users expect them to be responsive quickly after waking. I hope we can support this inside tokio.
We've tried to work around it, but without
timeoutsupport forCLOCK_BOOTTIMEor similar, our workaround is incomplete.
What restrictions did you run into? Would it not be possible to create a future that expires after X time with respect to the boot time and then to replace all timeout uses by uses that wait for either the actual future or your custom timeout future? Of course that would require you to write a significant amount of code if you want to be performant and cross platform but I don't think it should be impossible to do this outside of tokio.
Regarding changing tokio::time to use CLOCK_BOOTTIME, the main challenge is that although Tokio uses its own Instant type to support pausing time in tests, it has conversions to and from the std Instant type, and it's very unclear what those conversions should do if you change Tokio's Instant type to use CLOCK_BOOTTIME if the std Instant isn't also changed.
If you just want a separate set of timers that work with CLOCK_BOOTTIME, then that would be pretty easy since you can use a timerfd via Tokio's AsyncFd type.
Just an update: It seems like std is going to switch to using CLOCK_BOOTTIME in https://github.com/rust-lang/rust/pull/88714. We will need to implement support for that in Tokio by changing the timer driver to use a timerfd for timeouts. I think a reasonable plan is to implement that PR once the change is available in nightly so we can test it. We can hold off on merging it until it arrives in stable.
https://github.com/rust-lang/rust/pull/88714 got closed.
There is an upstream bug report for Rust at https://github.com/rust-lang/rust/issues/71860
While I am of the opinion that
tokio::timeshould be converted to useCLOCK_BOOTTIME(including thetimerfdusage), I could also see creating a paralleltokio::boot_timemodule or similar in order to expose this functionality without a change in behavior if we have some reason to believe that someone might be depending onCLOCK_MONOTONICbehavior.
Go tried to switch from CLOCK_MONOTINC to CLOCK_BOOTTIME behavior by default (at least on Windows) and had to revert this change as it broke existing usecases such as watchdog timers. Current accepted proposal is that there should be a second type of timestamps and timeouts: https://github.com/golang/go/issues/36141 I don't see any progress on the implementation though.
I suggest to copy Go decision and provide new type of timers instead of considering trying to change existing ones.
- On linux, iouring only supports the monotonic clock.
io_uring provides Timeout operation that has CLOCK_BOOTIME flag.
