bcachefs
bcachefs copied to clipboard
koverstreet
Filesystem stuck not freeing space
bcachefs fs usage -ah
Filesystem: 2c9b10da-e32f-44ab-b303-d2cd1005acf2
Size: 3.46 TiB
Used: 3.45 TiB
Online reserved: 0 B
Data by durability desired and amount degraded:
undegraded
1x: 3.36 TiB
2x: 116 GiB
cached: 35.0 GiB
reserved: 318 MiB
Data type Required/total Durability Devices
reserved: 1/1 [] 318 MiB
btree: 1/2 2 [dm-17 dm-16] 116 GiB
user: 1/1 1 [dm-17] 3.36 TiB
cached: 1/1 1 [dm-16] 35.0 GiB
Compression:
type compressed uncompressed average extent size
lz4 13.0 GiB 27.8 GiB 55.6 KiB
zstd 268 GiB 783 GiB 51.5 KiB
incompressible 3.13 TiB 3.13 TiB 55.6 KiB
Btree usage:
extents: 52.4 GiB
inodes: 35.0 GiB
dirents: 4.43 GiB
xattrs: 6.00 MiB
alloc: 523 MiB
quotas: 512 KiB
stripes: 512 KiB
reflink: 344 MiB
subvolumes: 512 KiB
snapshots: 3.00 MiB
lru: 8.00 MiB
freespace: 2.50 MiB
need_discard: 1.00 MiB
backpointers: 12.5 GiB
bucket_gens: 7.00 MiB
snapshot_trees: 512 KiB
deleted_inodes: 512 KiB
logged_ops: 512 KiB
reconcile_work: 1.00 MiB
subvolume_children: 512 KiB
accounting: 10.5 GiB
reconcile_scan: 56.0 MiB
Pending reconcile: data metadata
target: 0 B 1.25 MiB
pending: 0 B 57.9 GiB
hdd (device 0): dm-17 rw 94%
data buckets fragmented
free: 198 GiB 101160
sb: 3.00 MiB 3 3.00 MiB
journal: 8.00 GiB 4096
btree: 57.9 GiB 35184 10.9 GiB
user: 3.36 TiB 1767151 5.94 GiB
cached: 0 B 0
parity: 0 B 0
stripe: 0 B 0
need_gc_gens: 0 B 0
need_discard: 0 B 0
unstriped: 0 B 0
capacity: 3.64 TiB 1907594
bucket size: 2.00 MiB
ssd (device 1): dm-16 rw 73%
data buckets fragmented
free: 5.22 GiB 2672
sb: 3.00 MiB 3 3.00 MiB
journal: 1.00 GiB 512
btree: 57.9 GiB 40731 21.7 GiB
user: 0 B 0
cached: 35.0 GiB 21618 7.23 GiB
parity: 0 B 0
stripe: 0 B 0
need_gc_gens: 0 B 0
need_discard: 0 B 0
unstriped: 0 B 0
capacity: 128 GiB 65536
bucket size: 2.00 MiB
bcachefs fs usage -h
Filesystem: 2c9b10da-e32f-44ab-b303-d2cd1005acf2
Size: 3.46 TiB
Used: 3.45 TiB
Online reserved: 0 B
Data by durability desired and amount degraded:
undegraded
1x: 3.36 TiB
2x: 116 GiB
cached: 35.0 GiB
reserved: 318 MiB
Pending reconcile: data metadata
target: 0 B 1.25 MiB
pending: 0 B 57.9 GiB
Device label Device State Size Used Use% Leaving
hdd (device 0): dm-17 rw 3.64 TiB 3.43 TiB 94% 1.25 MiB
ssd (device 1): dm-16 rw 128 GiB 93.8 GiB 73%
bcachefs show-super
External UUID: 2c9b10da-e32f-44ab-b303-d2cd1005acf2
Internal UUID: ee332d9f-f329-4f7f-8037-91cea1c3bdda
Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index: 0
Label: Home
Version: reconcile (1.33)
Incompatible features allowed: reconcile (1.33)
Incompatible features in use: reconcile (1.33)
Version upgrade complete: reconcile (1.33)
Oldest version on disk: extent_flags (1.25)
Created: Wed Apr 30 09:15:57 2025
Sequence number: 11099
Time of last write: Thu Nov 20 22:56:52 2025
Superblock size: 6.79 KiB/1.00 MiB
Clean: 0
Devices: 2
Sections: crypt,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade,recovery_passes,extent_type_u64s
Features: lz4,zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes,incompat_version_field,casefolding
Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done
Options:
block_size: 4.00 KiB
btree_node_size: 256 KiB
errors: continue [fix_safe] panic ro
write_error_timeout: 30
metadata_replicas: 2
data_replicas: 1
metadata_replicas_required: 1
data_replicas_required: 1
encoded_extent_max: 64.0 KiB
metadata_checksum: none crc32c crc64 [xxhash]
data_checksum: none crc32c crc64 [xxhash]
checksum_err_retry_nr: 3
compression: lz4
background_compression: zstd
str_hash: crc32c crc64 [siphash]
metadata_target: none
foreground_target: ssd
background_target: hdd
promote_target: ssd
erasure_code: 0
casefold: 0
inodes_32bit: 1
shard_inode_numbers_bits: 4
inodes_use_key_cache: 1
gc_reserve_percent: 8
gc_reserve_bytes: 0 B
root_reserve_percent: 0
wide_macs: 0
promote_whole_extents: 1
acl: 1
usrquota: 0
grpquota: 0
prjquota: 0
degraded: [ask] yes very no
journal_flush_delay: 1000
journal_flush_disabled: 0
journal_reclaim_delay: 100
journal_transaction_names: 1
allocator_stuck_timeout: 30
version_upgrade: [compatible] incompatible none
nocow: 0
reconcile_on_ac_only: 0
errors (size 232):
reconcile_bp_to_missing_btree_ptr 89916 Thu Nov 20 18:35:10 2025
btree_ptr_to_bad_reconcile_bp 89807 Thu Nov 20 18:35:10 2025
inode_i_sectors_wrong 4 Thu Nov 20 17:47:46 2025
inode_dir_has_nonzero_i_size 25879639 Wed Nov 19 22:20:52 2025
inode_parent_has_case_insensitive_not_set 5 Wed Nov 19 22:09:28 2025
extent_io_opts_not_set 9 Wed Nov 19 11:48:08 2025
accounting_mismatch 7 Tue Nov 18 17:59:55 2025
accounting_key_underflow 2 Tue Nov 18 17:54:35 2025
vfs_inode_i_blocks_underflow 68 Tue Nov 18 15:14:04 2025
inode_i_sectors_underflow 72 Tue Nov 18 15:14:04 2025
reconcile_work_incorrectly_set 7499 Mon Nov 17 19:05:54 2025
inode_wrong_backpointer 2 Tue Sep 9 08:20:06 2025
vfs_bad_inode_rm 600 Mon Sep 8 08:47:57 2025
lru_entry_bad 1 Fri May 23 08:47:01 2025
ext (size 96):
Recovery passes required:
Errors to silently fix:
Btrees with missing data:
Device 0: /dev/dm-17 (unknown model)
Label: hdd
UUID: 6b61a3e6-529b-4e09-a7c1-84c7670f64b0
Size: 3.64 TiB
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00 MiB
First bucket: 0
Buckets: 1907594
Last mount: Thu Nov 20 22:56:49 2025
Last superblock write: 11099
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,user
Btree allocated bitmap blocksize: 128 MiB
Btree allocated bitmap: 0000000000000000000000000000001100000010000000100000001010010111
Durability: 1
Discard: 0
Freespace initialized: 1
Resize on mount: 0
Last device name: dm-17
Last device model:
Device 1: /dev/dm-16 (unknown model)
Label: ssd
UUID: dff9852b-6fcc-44fb-b0b8-73b278cbffad
Size: 128 GiB
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00 MiB
First bucket: 0
Buckets: 65536
Last mount: Thu Nov 20 22:56:49 2025
Last superblock write: 11099
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,cached
Btree allocated bitmap blocksize: 4.00 MiB
Btree allocated bitmap: 0011111111111111111111111111111111111111111111111111111111111111
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Last device name: dm-16
Last device model:
nothing looks off here, possibly space being taken up by snapshots you said - so we can't do anything with this until we have per-snapshot accounting exposed
Attempting to summarize from IRC: the fs has metadata_replicas=2 (metadata_replicas_required=1), my intent is to have one replica on ssd and one on hdd. There is tiering: foreground_target=ssd, background_target=hdd, promote_target=ssd, but I don't intend it to be strict. I might be able to work around this by adding another hdd device backed by another disk, but don't have a spare ssd. Or by explicitly configuring metadata to land anywhere; metadata_replicas would ensure one copy is on ssd and have the same effect on read performance as explicit tiering.
ETA: this seemed to work as expected (no ENOSPC issues that couldn't be fixed by freeing some space) pre-reconcile.
nothing looks off here, possibly space being taken up by snapshots you said - so we can't do anything with this until we have per-snapshot accounting exposed
I've deleted stuff (at least 200 GB IIRC) across all snapshots, but was wondering if already deleted snapshots truly were.
Oh, should have added the important part:
LC_ALL=C df ~
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/Consolidated/Home:/dev/Consolidated/HomeCache 3708243148 3708243148 0 100% /home/g2p
which is completely unchanging, despite file deletions (across the main volume and all snapshots). So accounting seems stuck.
/sys/kernel/debug/bcachefs/2c9b10da-e32f-44ab-b303-d2cd1005acf2/btrees/snapshots/keys has 3200 lines which seems high, 2314 grep for delete, 993 for no_keys, 123 match neither. The number I would expect is ls ~/.snapshots |wc -l which is 65. There are rotating snapshots created and deleted every hour, through minisnap (msnap).
https://sebsauvage.net/paste/?6d2d2cf98656d386#EhicFvZY7FQQeY8ak2G2MmU5ZXFcrk5/scRkh2M0h0M=
We don't delete snapshot keys for deleted snapshots, we tombstone them - check snapshot_delete_status
We don't delete snapshot keys for deleted snapshots, we tombstone them - check snapshot_delete_status
(not running), so, I have to look elsewhere
I deleted a snapshot, checked it was ongoing in snapshot_delete_status, monitored with fs top
and perf trace -e 'bcachefs:error_throw'.
0.000 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
0.032 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
188.680 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
188.707 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
429.203 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
429.234 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
526.755 kswapd0/118 bcachefs:error_throw(dev: 265289745, err: 4294965186, err_str: "ENOMEM_btree_node_reclaim", ip: "__btree_node_reclaim_checks")
526.817 kswapd0/118 bcachefs:error_throw(dev: 265289745, err: 4294965186, err_str: "ENOMEM_btree_node_reclaim", ip: "__btree_node_reclaim_checks")
610.606 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
610.633 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1215.134 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
1215.168 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1383.417 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
1383.458 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1511.271 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
1511.298 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1799.000 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
1799.023 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1860.805 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
1860.831 kworker/9:1+bc/41718 bcachefs:error_throw(dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
2362.375 kswapd0/118 bcachefs:error_throw(dev: 265289745, err: 4294965186, err_str: "ENOMEM_btree_node_reclaim", ip: "__btree_node_reclaim_checks")
2564.165 bch-reclaim/dm/29065 bcachefs:error_throw(dev: 265289740, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
sed |uniq -c
1 (dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
1 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1 (dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
3 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
1 (dev: 265289745, err: 4294965094, err_str: "transaction_restart_write_buffe", ip: "bch2_btree_write_buffer_maybe_f")
1 (dev: 265289745, err: 4294965076, err_str: "backpointer_to_overwritten_btre", ip: "__bch2_backpointer_get_node")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
8 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1 (dev: 265289745, err: 4294965094, err_str: "transaction_restart_write_buffe", ip: "bch2_btree_write_buffer_maybe_f")
1 (dev: 265289745, err: 4294965076, err_str: "backpointer_to_overwritten_btre", ip: "__bch2_backpointer_get_node")
6 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
1 (dev: 265289745, err: 4294965094, err_str: "transaction_restart_write_buffe", ip: "bch2_btree_write_buffer_maybe_f")
1 (dev: 265289745, err: 4294965076, err_str: "backpointer_to_overwritten_btre", ip: "__bch2_backpointer_get_node")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
3 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
1 (dev: 265289745, err: 4294965094, err_str: "transaction_restart_write_buffe", ip: "bch2_btree_write_buffer_maybe_f")
1 (dev: 265289745, err: 4294965076, err_str: "backpointer_to_overwritten_btre", ip: "__bch2_backpointer_get_node")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
2 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
2 (dev: 265289745, err: 4294965094, err_str: "transaction_restart_write_buffe", ip: "bch2_btree_write_buffer_maybe_f")
4 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1 (dev: 265289745, err: 4294965094, err_str: "transaction_restart_write_buffe", ip: "bch2_btree_write_buffer_maybe_f")
1 (dev: 265289745, err: 4294965076, err_str: "backpointer_to_overwritten_btre", ip: "__bch2_backpointer_get_node")
2 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1 (dev: 265289745, err: 4294965094, err_str: "transaction_restart_write_buffe", ip: "bch2_btree_write_buffer_maybe_f")
1 (dev: 265289745, err: 4294965076, err_str: "backpointer_to_overwritten_btre", ip: "__bch2_backpointer_get_node")
8 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
1 (dev: 265289745, err: 4294965094, err_str: "transaction_restart_write_buffe", ip: "bch2_btree_write_buffer_maybe_f")
1 (dev: 265289745, err: 4294965076, err_str: "backpointer_to_overwritten_btre", ip: "__bch2_backpointer_get_node")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
2 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
1 (dev: 265289745, err: 4294965094, err_str: "transaction_restart_write_buffe", ip: "bch2_btree_write_buffer_maybe_f")
1 (dev: 265289745, err: 4294965076, err_str: "backpointer_to_overwritten_btre", ip: "__bch2_backpointer_get_node")
3 (dev: 265289745, err: 4294964992, err_str: "journal_blocked", ip: "__journal_res_get")
1 (dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
1 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1 (dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
1 (dev: 265289745, err: 4294965185, err_str: "ENOMEM_btree_node_mem_alloc", ip: "__bch2_err_trace")
1 (dev: 265289745, err: 4294964998, err_str: "insufficient_devices", ip: "bch2_bucket_alloc_set_trans")
1 (dev: 265289745, err: 4294965185, err_str: "ENOMEM_btree_node_mem_alloc", ip: "__bch2_err_trace")
1 (dev: 265289745, err: 4294965080, err_str: "btree_insert_btree_node_full", ip: "__bch2_trans_commit")
2s total mount
io_read 0 832 3144936752
io_read_inline 0 0 5823637
io_read_hole 0 0 510552
io_read_nopromote 0 18 29125594
io_read_nopromote_may_not 0 0 5234
io_read_nopromote_already_promoted 0 18 652378
io_read_nopromote_congested 0 0 35667
io_read_nopromote_in_flight 0 0 6
io_read_bounce 0 18 3239426
io_read_split 0 0 9010570
io_write 0 0 8
data_update 0 0 5234
data_update_fail 0 0 28432304
data_update_key 0 0 420024
data_update_key_fail 0 0 39
io_move_read 0 0 272520
io_move_evacuate_bucket 0 192 7475922
rebalance_extent 0 0 104404
bucket_invalidate 0 12 312
bucket_discard 0 378 7138
bucket_discard_fast 0 0 94
bucket_alloc 0 460 6979
btree_cache_scan 11 842 29381
btree_cache_reap 697 76525 1668347
btree_cache_cannibalize_lock 0 1202 21423
btree_cache_cannibalize_lock_fail 0 0 33
btree_cache_cannibalize_unlock 0 1202 21423
btree_node_write 6 32247 608653
btree_node_read 747 76058 1701138
btree_node_compact 0 261 8890
btree_node_merge 0 0 1279
btree_node_split 0 622 6087
btree_node_rewrite 0 340 5310
btree_node_alloc 0 1845 27653
btree_node_free 0 1468 32397
btree_node_set_root 0 1 51
btree_path_relock_fail 0 1914 32027
btree_path_upgrade_fail 0 974 6700
btree_reserve_get_fail 0 1 55
journal_full 0 0 111
journal_reclaim_finish 18 35242 2268094
journal_reclaim_start 18 35242 2268094
journal_write 2 1458 58338
copygc 0 12 467241
copygc_wait 0 24 238
trans_restart_btree_node_split 0 3 146
trans_restart_mem_realloced 0 0 60
trans_restart_relock 0 16 861
trans_restart_relock_path 751 76501 1887038
trans_restart_upgrade 0 710 5170
trans_restart_would_deadlock 0 0 475
trans_traverse_all 745 77036 1772356
transaction_commit 9 5681619 57117104
write_super 0 1 54
trans_restart_write_buffer_flush 0 75 809
write_buffer_flush_slowpath 0 0 11
write_buffer_flush_sync 0 137 1302
accounting_key_to_wb_slowpath 0 812 12751
error_throw 712 33964 64133030