awesome-prometheus-alerts icon indicating copy to clipboard operation
awesome-prometheus-alerts copied to clipboard

Rule "Host out of inodes" triggers false positive with FAT32 on FreeBSD

Open SamK opened this issue 1 year ago • 2 comments

Hi,

I'm monitoring my OPNsense box with Node exporter. The rule HostOutOfInodes triggers an error with this result:

{device="/dev/gpt/efiboot0", fstype="msdosfs", instance="my-opnsense-box:9100", job="node_exporter", mountpoint="/boot/efi", nodename="opn"} = 0

It should not trigger that error. I tried to think about a generic way to resolve the query but I think the msdosfs fstype should be simply excluded from the query.

I can provide a PR.

SamK avatar Jan 19 '24 13:01 SamK

Can you develop please? I'm mostly a linux+macos user. In what situation is msdosfs used?

samber avatar Jan 20 '24 17:01 samber

I have this exact issue on my OPNSense firewall that is freebsd based.

AnderssonPeter avatar Jan 31 '24 15:01 AnderssonPeter

The FAT16 file system is used for the EFI partition. FAT file systems do not have such thing as inodes. However FreeBSD returns a value of 512 inodes for FAT16. This is a behaviour explained in their mailing list, described by @stesser:

[...] there is no limit on the number of files (reported as inodes) in the root directory of a FAT32 file system, but such a limit does exist in FAT12/FAT16 and the default value happens to be 512 in case of FAT16.

My partition is indeed FAT16 and I am victim of this bug. ifree below should not be 0.

# df -i /boot/efi
Filesystem        1K-blocks Used  Avail Capacity iused ifree %iused  Mounted on
/dev/gpt/efiboot0    266176 1872 264304     1%     512     0  100%   /boot/efi
# file -s /dev/gpt/efiboot0
/dev/gpt/efiboot0: DOS/MBR boot sector, code offset 0x3c+2, OEM-ID "BSD4.4  ", sectors/cluster 32, root entries 512, sectors/FAT 65, sectors/track 63, heads 16, sectors 532480 (volumes > 32 MB), serial number 0xf08b1618, unlabeled, FAT (16 bit)
  • The bug is fixed in FreeBSD 13.3.
  • As of today, OPNSense 24.1 is based on FreeBSD 13.2.
  • The OPNSense roadmap has not yet scheduled to use FreeBSD 13.3.

I hope this explains the situation.

@samber, this bug will be auto-fixed in an unknown future. In the meantime a PR can easily fix it today. As the maintainer of this project it's up to you to choose what decision to take.

SamK avatar Feb 07 '24 14:02 SamK

The MSDOSFS code in 13.3. could be backported to 13.2, but a simpler work-around would be to just return 0 for the total number of inodes (f_files). The value 0 is typically used for synthetic file systems (procfs, devfs, ...) to indicate that they do not actually use inodes (and AFAIK, monitoring systems understand this signal and ignore the inode stats).

diff --git a/sys/fs/msdosfs/msdosfs_vfsops.c b/sys/fs/msdosfs/msdosfs_vfsops.c
index d7efa103581b..ba73a1f18ce4 100644
--- a/sys/fs/msdosfs/msdosfs_vfsops.c
+++ b/sys/fs/msdosfs/msdosfs_vfsops.c
@@ -950,8 +950,8 @@ msdosfs_statfs(struct mount *mp, struct statfs *sbp)
        sbp->f_blocks = pmp->pm_maxcluster + 1;
        sbp->f_bfree = pmp->pm_freeclustercount;
        sbp->f_bavail = pmp->pm_freeclustercount;
-       sbp->f_files = pmp->pm_RootDirEnts;     /* XXX */
-       sbp->f_ffree = 0;       /* what to put in here? */
+       sbp->f_files = 0;
+       sbp->f_ffree = 0;
        return (0);
 }

If this simple patch was applied to OPNSense, the "out of inodes" issue should be resolved.

(A similar patch should be applied to all synthetic file systems, some of which also reported 100% inodes used. This is fixed in 13.3 and later versions, too.)

stesser avatar Feb 07 '24 16:02 stesser

I just made a fix and prepared a revert for 2025.

samber avatar Feb 10 '24 19:02 samber

The commit fixes the issue. Thank you. I'm taking the liberty to close the issue. Best regards.

SamK avatar Feb 16 '24 12:02 SamK