rdedup icon indicating copy to clipboard operation
rdedup copied to clipboard

Published 20 hours ago • verified publisher icon dpc

Can disk IO be improved any more?

Open dpc opened this issue 6 years ago • 2 comments

  • writev - doesn't seem to make a difference
  • fallocate - maybe? blocked on https://github.com/nix-rust/nix/issues/596
  • O_DIRECT - to prevent cache pollution? or fadvise?

dpc avatar Sep 10 '17 19:09 dpc

Especially on spinning disks and if you know the order you will be traversing files it can help to run a mmap(..., MADV_WILLNEED) on files before the input stage reads them the first time.

The trick is figuring out how far ahead of the main process to run. This can backfire if you are memory limited.

wscott avatar Feb 03 '18 12:02 wscott

Good tip.

Right now I'm not planing to look into it, but if anyone benefits, please give it a try.

dpc avatar Feb 03 '18 21:02 dpc