[Bug] FSCache on demand loading not working

[singh-saurabh]

Problem Description

When running nydus-snapshotter with --fs-driver fscache, fscache appears to initialize and cache data, but container file reads never hit the cache.

This internally shows

INFO[2025-09-18T12:04:31.708240749Z] Start nydus-snapshotter. Version: v0.15.3, PID: 7849, FsDriver: fscache, DaemonMode: multiple
INFO[2025-09-18T12:04:31.710484604Z] Run daemons monitor...
INFO[2025-09-18T12:04:31.710717679Z] initializing shared nydus daemon for fscache
INFO[2025-09-18T12:04:31.713075176Z] nydusd command: /usr/local/bin/nydusd singleton --fscache /var/lib/containerd/io.containerd.snapshotter.v1.nydus/cache --apisock /var/lib/containerd/io.containerd.snapshotter.v1.nydus/socket/d35vajrt64uh53eit6q0/api.sock --log-level info --log-rotation-size 100
[2025-09-18 12:04:31.714596 +00:00] INFO Program Version: v2.3.6, Git Commit: "cb50bd934426d3532534432426c4a701ecaec4a1", Build Time: "2025-09-09T10:40:34.901064076Z", Profile: "release", Rustc Version: "rustc 1.84.0 (9fc6b4312 2025-01-07)"

Observations

1. Fscache infrastructure appears functional: Cache writes (167k operations) prove fscache subsystem is working
2. EROFS mounts successfully: Kernel logs show erofs mounting with fscache detection
3. Zero cache reads: Despite cache writes, no read operations hit the cache
4. Likely Result: All container file reads go to backend storage instead of cache (unsure)

Expected Behavior

Container file reads should hit fscache, showing cache read statistics This should result in performance better than Overlay/FUSE mode

Actual Behavior

All container file reads bypass cache, going directly to backend storage. Performance worse than Nydus+FUSE and equal to Overlay.

cat /proc/fs/fscache/stats

Reads  : DR=0 RA=0 RF=0 RS=0 WB=0 WBZ=0
Writes : BW=0 WT=0 DW=0 WP=0 2C=0
ZeroOps: ZR=0 sh=0 sk=0
DownOps: DL=0 ds=0 df=0 di=0
CaRdOps: RD=0 rs=0 rf=0
UpldOps: UL=0 us=0 uf=0
CaWrOps: WR=0 ws=0 wf=0
Retries: rq=0 rs=0 wq=0 ws=0
Objs   : rr=0 sr=0 foq=0 wsc=0
WbLock : skip=0 wait=0
-- FS-Cache statistics --
Cookies: n=0 v=0 vcol=0 voom=0
Acquire: n=100 ok=100 oom=0
LRU    : n=0 exp=0 rmv=0 drp=100 at=0
Invals : n=0
Updates: n=0 rsz=0 rsn=0
Relinqs: n=100 rtr=0 drop=100
NoSpace: nwr=0 ncr=0 cull=73
IO     : rd=46058 wr=169765 mis=0

Kernel Logs Show Fscache Detection

kernel: netfs: Cache "CacheFiles" added (type cachefiles)
kernel: CacheFiles: File cache on CacheFiles registered
kernel: erofs: [deprecated] fscache-based on-demand read feature in use.

How to reproduce

1. Run sudo containerd-nydus-grpc --nydusd-config /etc/nydus/nydusd-config.fscache.json --fs-driver fscache --log-to-stdout with the config I provided
2. Run sudo nerdctl run --rm -it --snapshotter=nydus docker.io/hsiangkao/ubuntu:20.04-rafs-v6 /bin/bash
3. Check /proc/fs/fscache/stats

Environment Details

• Nydus-snapshotter version: v0.15.3
• Nydus version: v2.3.6
• Container runtime:
• Operating System: Ubuntu 24.04.3 LTS
• Kernel version: 6.14.11
• Mount shows: erofs ... (ro,relatime,user_xattr,acl,cache_strategy=readaround,fsid=...)

Nydusd config:

{
	"type": "bootstrap",
	"config": {
		"backend_type": "registry",
		"backend_config": {},
		"cache_type": "fscache",
		"prefetch_config": {
			"enable": false,
			"threads_count": 16,
			"merging_size": 1048576
		}
	}
}

Additional Information

• Fscache provides no performance benefit (0% cache hit rate)
• All container file reads go to backend storage
• On-demand loading not working as expected

Are you willing to submit PR?

• [x] Yes I am willing to submit a PR!

[imeoer]

kernel: erofs: [deprecated] fscache-based on-demand read feature in use.

Is fscache support dropped from EROFS? @hsiangkao

[hsiangkao]

kernel: erofs: [deprecated] fscache-based on-demand read feature in use.

Is fscache support dropped from EROFS? @hsiangkao

As the kernel message shown, recent kernels deprecate this feature because it will have a cleaner alternative, but erofs fscache feature can still be used until the latest kernel (v6.17) for now.

[PKizzle]

Will nydus support fanotify pre-content hooks then? https://kernel.googlesource.com/pub/scm/linux/kernel/git/xiang/linux/+/refs/heads/erofs/remove_fscache