pcircle icon indicating copy to clipboard operation
pcircle copied to clipboard

Optimize fprof for Lustre attribute retrieval

Open adilger opened this issue 6 years ago • 1 comments

The "lfs getstripe" and "lfs find" code is using an ioctl(LL_IOC_MDC_GETINFO) to fetch both the stat attributes and Lustre layout from the MDS in a single RPC. This also opens only the parent directory for the ioctl(), so that each file does not need to be opened (which avoids inode cache pollution and DLM lock/cancel overhead as well).

With the "Lazy Size-on-MDT" (LSOM) feature that is being added in 2.12 it would also be possible to fetch a reasonable estimate of the file size from the MDS as well, which would avoid the need for any OSS RPCs entirely, which can add significant overhead if the MDT(s) are SSDs and the OSTs are HDDs. For the purposes of fprof (histograms and average file sizes), the LSOM size is sufficient.

adilger avatar May 08 '18 01:05 adilger

thanks for the suggestion, Andreas, I will look into it.

Feiyi

fwang2 avatar Jun 23 '18 10:06 fwang2