dCache
leftover empty data files on pools
Hey.
For quite a while I suffer from the btrfs issue I've vaguely described in #5352.
What happens is basically, that btrfs allocates the storage in block groups some for meta-data some for data. It may now happen that all free block groups are allocated to either of the two, that meta-data block groups are full but data block groups are not.
Thus df like free-space-detection would still show there's plenty of free space left, whereas any writes would still fail as no new meta-data can be written.
I never really found out which IO patterns exactly produce that (cause e.g. I never see it on my private btrfs filesystems).
By chance I might have found one candidate, probably not the main cause, because on that particular pool the situation was vice versa, i.e. meta-data block groups were not yet full, but data was.
It seems dCache still tried to write new files to the pool, but the actual writing always failed (see billings below), however a fs entry was created (because meta-data wasn't full yet), all empty files, all from today, several hundreds.
Checking the PNFSIDs of these files with pathfinder they’re not in the DB.
But dCache doesn't seem to clean up the empty files on the pool either, which is bad.
Cheers, Chris.
