Hierarchical bandwidth and operations rate limits.

[pjd]

Introduce six new properties: limit_{bw,op}_{read,write,total}.

The limit_bw_* properties limit the read, write, or combined bandwidth, respectively, that a dataset and its descendants can consume. Limits are applied to both file systems and ZFS volumes.

The configured limits are hierarchical, just like quotas; i.e., even if a higher limit is configured on the child dataset, the parent's lower limit will be enforced.

The limits are applied at the VFS level, not at the disk level. The dataset is charged for each operation even if no disk access is required (e.g., due to caching, compression, deduplication, or NOP writes) or if the operation will cause more traffic (due to the copies property, mirroring, or RAIDZ).

Read bandwidth consumption is based on:

• read-like syscalls, eg., aio_read(2), pread(2), preadv(2), read(2), readv(2), sendfile(2)

• syscalls like getdents(2) and getdirentries(2)

• reading via mmaped files

• zfs send

Write bandwidth consumption is based on:

• write-like syscalls, eg., aio_write(2), pwrite(2), pwritev(2), write(2), writev(2)

• writing via mmaped files

• zfs receive

The limit_op_* properties limit the read, write, or both metadata operations, respectively, that dataset and its descendants can generate.

Read operations consumption is based on:

• read-like syscalls where the number of operations is equal to the number of blocks being read (never less than 1)

• reading via mmaped files, where the number of operations is equal to the number of pages being read (never less than 1)

• syscalls accessing metadata: readlink(2), stat(2)

Write operations consumption is based on:

• write-like syscalls where the number of operations is equal to the number of blocks being written (never less than 1)

• writing via mmaped files, where the number of operations is equal to the number of pages being written (never less than 1)

• syscalls modifing a directory's content: bind(2) (UNIX-domain sockets), link(2), mkdir(2), mkfifo(2), mknod(2), open(2) (file creation), rename(2), rmdir(2), symlink(2), unlink(2)

• syscalls modifing metadata: chflags(2), chmod(2), chown(2), utimes(2)

• updating the access time of a file when reading it

Just like limit_bw_* limits, the limit_op_* limits are also hierarchical and applied at the VFS level.

Motivation and Context

Description

How Has This Been Tested?

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Performance enhancement (non-breaking change which improves efficiency)
• [ ] Code cleanup (non-breaking change which makes code smaller or more readable)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)
• [ ] Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• [ ] Documentation (a change to man pages or other documentation)

Checklist:

• [x] My code follows the OpenZFS code style requirements.
• [x] I have updated the documentation accordingly.
• [x] I have read the contributing document.
• [x] I have added tests to cover my changes.
• [ ] I have run the ZFS Test Suite with this change applied.
• [x] All commit messages are properly formatted and contain Signed-off-by.

[tonyhutter]

Some first-pass questions/comments:

Maybe I missed it, but do you specify the units anywhere? Are the limit_bw_* props in units of bytes/second and limit_op_* in ops/second? Can you mention the units in the man pages?

If you specify 100MB/s and they system is idle with 8GB/s bandwidth available, will it still only use 100MB/s?

What happens if someone specifies a *_total value that is larger/smaller than *_read or *_write combined? Does it just get capped to the minimum?

Does anything bad happen if you cap it to something super low, like 1 byte/sec?

The zfsprops.7 man page is roughly in alphabetical order. Could you move the limits_* section to just after the keylocation sections?

Is copy_file_range() counted?

[pjd]

Maybe I missed it, but do you specify the units anywhere? Are the limit_bw_* props in units of bytes/second and limit_op_* in ops/second? Can you mention the units in the man pages?

Sure, I'll add that.

If you specify 100MB/s and they system is idle with 8GB/s bandwidth available, will it still only use 100MB/s?

Correct.

What happens if someone specifies a *_total value that is larger/smaller than *_read or *_write combined? Does it just get capped to the minimum?

Correct. Always the lowest limit will be enforced. The same if the parent has a lower limit then the child or children combined.

Does anything bad happen if you cap it to something super low, like 1 byte/sec?

It cannot be lower than the resolution (which is 16 per second), so it will be rounded up to 16, but it will also allocate large number of slots to keep the history, so here one slot per byte of each pending request.

The zfsprops.7 man page is roughly in alphabetical order. Could you move the limits_* section to just after the keylocation sections?

Sure.

Is copy_file_range() counted?

So it was counted when it was doing fall back to read/write, but it wasn't counted in case of block cloning, which I think it should, so I just added it.

[tonyhutter]

I did some hand testing of this and it works just as described :+1:

$ ./zpool create -f tank -O compression=off /dev/nvme{0..8}n1 && dd if=/dev/zero of=/tank/bigfile conv=sync bs=1M count=10000 && ./zpool destroy tank 
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 3.91677 s, 2.7 GB/s


$ ./zpool create -f tank -O compression=off /dev/nvme{0..8}n1 && ./zfs set limit_bw_write=$((1024 * 1024 * 200)) tank && dd if=/dev/zero of=/tank/bigfile conv=sync bs=1M count=10000 && ./zpool destroy tank
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 50.9869 s, 206 MB/s

I also verified that it worked for multithreaded writes, and verified that top-level dataset's values correctly overrode their children's values.

[pjd]

I did some hand testing of this and it works just as described 👍

$ ./zpool create -f tank -O compression=off /dev/nvme{0..8}n1 && dd if=/dev/zero of=/tank/bigfile conv=sync bs=1M count=10000 && ./zpool destroy tank 
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 3.91677 s, 2.7 GB/s


$ ./zpool create -f tank -O compression=off /dev/nvme{0..8}n1 && ./zfs set limit_bw_write=$((1024 * 1024 * 200)) tank && dd if=/dev/zero of=/tank/bigfile conv=sync bs=1M count=10000 && ./zpool destroy tank
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB, 9.8 GiB) copied, 50.9869 s, 206 MB/s

I also verified that it worked for multithreaded writes, and verified that top-level dataset's values correctly overrode their children's values.

Thank you!

BTW. You can use suffixes got limit_bw_* properties, eg. limit_bw_write=200M

[amotin]

Does anything bad happen if you cap it to something super low, like 1 byte/sec?

It cannot be lower than the resolution (which is 16 per second), so it will be rounded up to 16, but it will also allocate large number of slots to keep the history, so here one slot per byte of each pending request.

I am not sure what was original Tony's concern, but my first thought was that some 1MB write operation may stuck in kernel for 11 days. I am only starting to read the code, and I only hope that the wait is interruptible, otherwise it may become an administrative nightmare.

[pjd]

Does anything bad happen if you cap it to something super low, like 1 byte/sec?

It cannot be lower than the resolution (which is 16 per second), so it will be rounded up to 16, but it will also allocate large number of slots to keep the history, so here one slot per byte of each pending request.

I am not sure what was original Tony's concern, but my first thought was that some 1MB write operation may stuck in kernel for 11 days. I am only starting to read the code, and I only hope that the wait is interruptible, otherwise it may become an administrative nightmare.

It was not interruptible on purpose initially as I assumed it will be easy to generate more traffic than configured, but as long as we do accounting before we go to sleep we should be fine. There are few cases were I do accounting after the fact, but those are not the main ones. I'll make it interruptible.

[snajpa]

While some way to limit IO with OpenZFS is badly needed, I think this is not going to be very useful for setups that put the advantages of the ARC and the writeback buffering + optimizations towards the user. I'd like to communicate an idea rather than any negativity, this is awesome work and with the world moving on to NVMe only setups, it could get a pass even in our lower-cost environment eventually :)

There's the obvious catch with pairing ZIOs to objsets, but I'm thinking, can't that be solved using the SPL's tsd, thread specific data? You've done the hardest parts I didn't want to do (identifying accounting points + all the infrastructure for the limits management) to get to the point to be able to actually prove this idea, so I might just go and try that on top of whatever eventually lands; but if you have the time, you could probably try that way sooner before I get to it. The idea is to mark the thread when entering the ZPL with specific objset id in the top layers where that info is available and pull it out of tsd probably in zio_create directly to struct zio.

If you guys don't see why that approach won't work for any obvious reason, the very least we can do now put some thought how the UI should look like if we eventually get to have the ability to limit only L1/L2 read misses, synchronous writes, so it doesn't cause headaches later on... limit_{bw,op}_write_sync, limit_{bw,op}_read_l1arc_miss, limit_{bw,op}_read_l2arc_miss? I think that could work :)

[pjd]

If you guys don't see why that approach won't work for any obvious reason, the very least we can do now put some thought how the UI should look like if we eventually get to have the ability to limit only L1/L2 read misses, synchronous writes, so it doesn't cause headaches later on... limit_{bw,op}_write_sync, limit_{bw,op}_read_l1arc_miss, limit_{bw,op}_read_l2arc_miss? I think that could work :)

One of the goals for this design was to provide predictability. When you move benefits of the ARC to the user and we don't charge the user for what's in the ARC already it will be hard to tell what to expect. Also, if we do that, why not to move the benefits of deduplication, NOP writes, block cloning and compression to the user as well? And while we are here why not to charge the user for some extra costs, like write inflation caused by RAIDZ, mirroring or the copies property or even metadata? Not to mention that limiting writes too low is a problem when we are in the TXG syncing context as we don't want to slow down the whole transaction.

This design I think fits better to environments where most people would like to use it - where you share the storage among many independent consumers like VMs or containers. As a storage provider you get the benefits of compression, deduplication, etc. but you also handle the cost of various inflations that would be hard to justify to the consumers. Also, with this design it is trivial for the consumer to test that they are getting the limits the see being configured.

[pjd]

Does anything bad happen if you cap it to something super low, like 1 byte/sec?

It cannot be lower than the resolution (which is 16 per second), so it will be rounded up to 16, but it will also allocate large number of slots to keep the history, so here one slot per byte of each pending request.

I am not sure what was original Tony's concern, but my first thought was that some 1MB write operation may stuck in kernel for 11 days. I am only starting to read the code, and I only hope that the wait is interruptible, otherwise it may become an administrative nightmare.

I reimplemented all the vfs_ratelimit functions to be interruptible.

[snajpa]

@pjd actually the use-case for this for us (vpsfree.org) is to ratelimit containers, which mirror the original Zones/Jails/OpenVZ motivations - there is an admin per each of those separate virtual environments - and they manage these containers as if they were a full VM (even running their own lxd + docker, etc., nested down).

A full VM would give those individual admins VFS caching, but if we limit VFS ops instead of IO ops, they're getting hit by a limit that isn't there when you run a full VM. Consider a static web serving a simple example - that relies on VFS caching heavily, a performance-wise sane design is to ensure frequently touched data fits in RAM. People usually don't have to resort to explicitly caching the contents (and metadata) of those files/directories in the userspace, the kernel does it for them well-enough to catch the gist of such use-cases.

Arguably, ZFS should be even better for these use-cases due to MRU/MFU split by design.

Now, when we're limiting IOPS or IO bandwidth, it's due to actual backing storage limitations, the only limitation for RAM there is - is its capacity. The backing devices are what's bandwidth/IOPS constrained and when thinking of QoS, this is where I'd like to ensure fairness the most. I see some of our containers pulling multiple GB/s from the ARC regularly in peak hours and I think that's wonderful they're able to do that - all while not touching the backing store. I don't think we can find a set of values for limits on the VFS layer to be able to really ensure fairness on the backend, while not severely limiting the frontend case (where data is already in RAM), when the difference between those two in absolute numbers is still this high.

Once the hardware levels up with RAM, then arguably, ARC is basically relevant only for caching decompressed/decrypted hot (meta)data, but that I still would like to push towards the admins of our containers, because that's still what they would get from a full VM, so it is what they're expecting.

While the backing-store QoS is made lvl-hard in ZFS due to the ZFS IO pipeline thread split, I think that is actually the area where it makes sense to focus on the most, given the general containers use-case especially. Even for docker/k8s/etc. Moreover, QEMU and other hypervisors can already do IO limiting on their own... so containers remain the primary concern - and they I think are used to their VFS cache being a cache.

And yes, being able to partition ARC or actually to have upper bounds on possible ARC usage for a single container, would be even better to have. And nice to have, too. But in practice, this has never been a concern, I only ever dug around the dbuf kstats out of curiosity (to see how much ARC does each container occupy and how that evolves during the day) - I thought that would be a concern for us too, but it wasn't in the end.

I can imagine it being a concern when the ARC is sized below the full workingset size. But frankly we like the ARC and are giving it 12.5-25% of system RAM on every box (12.5% on systems with >1T; 25% for 256G to 1T systems), so we get constantly awesome hitrates.

[snajpa]

Also, if we do that, why not to move the benefits of deduplication, NOP writes, block cloning and compression to the user as well?

To my best knowledge, these are all actually accounted in a way that benefits the user, already. For example, there is no quota I could set that would cap the uncompressed size, it's the compressed size that is under the quota (and that also gets presented to the userspace). Same goes for all the rest, I believe.

[allanjude]

Also, if we do that, why not to move the benefits of deduplication, NOP writes, block cloning and compression to the user as well?

To my best knowledge, these are all actually accounted in a way that benefits the user, already. For example, there is no quota I could set that would cap the uncompressed size, it's the compressed size that is under the quota (and that also gets presented to the userspace). Same goes for all the rest, I believe.

user quotas in ZFS are on the logical data, not the compressed (physical) data.

[snajpa]

@allanjude I've never used those and most people never do, those are user quotas where there are some preexisting expectations webhosters especially have... but OK, that's one area I didn't know about, all the rest is to benefit the user, not the provider, I'd say that user level quotas are a huge exception. Can you find another?

I think it would definitely be nice to have lsize-based quotas for datasets, FWIW.

But I also said I'd go and try to implement the idea with TSD marking + ZIO objset id propagation, I'm not saying anyone other than me should do the work, I wouldn't even dare to think that :) I'm just asking for some consideration, if we'd be okay with additional set of limits with suffixes (it's those _misses that are the most interesting to me)... and if anyone sees any obvious reason why that approach would be doomed even before I try :)

[amotin]

@pjd Aside of few remaining comments this needs a rebase according to github.

[youzhongyang]

By current code path, even if the iolimit properties are not defined / have default values, zfs_iolimit_*() functions need to do a whole bunch of things, this will definitely waste lots of CPU cycles when it's heavily loaded.

It would be nice if these zfs_iolimit_*() functions have a shortcut path / can quickly returns if no iolimit is defined, something like checking if dd_iolimit_root == NULL?

Another nice enhancement would be limiting I/O in container, i.e. iolimit can be defined and enforced per container. This is a more practical and useful use case, but it can be done after this PR is merged.

[pjd]

By current code path, even if the iolimit properties are not defined / have default values, zfs_iolimit_*() functions need to do a whole bunch of things, this will definitely waste lots of CPU cycles when it's heavily loaded.

It would be nice if these zfs_iolimit_*() functions have a shortcut path / can quickly returns if no iolimit is defined, something like checking if dd_iolimit_root == NULL?

That's good idea. I added the change.

Another nice enhancement would be limiting I/O in container, i.e. iolimit can be defined and enforced per container. This is a more practical and useful use case, but it can be done after this PR is merged.

This is harder. We sleep when we hit limits while holding locks (like the range lock), which is fine as long as it doesn't matter who accesses the file. All in all I think it is still usable with containers, as you probably want a dedicated dataset per container anyway.

[amotin]

This needs update after DMU_TXG_WAIT got renamed.