koverstreet
Possibly bcachefs caused corruption during Rust programs compilation.
For some time now (months) I'm sometimes experiencing corruption of Rust build artifacts. Typically it's Rust/linker complaining Unsupported version 0 of Verneed record about some intermediate rlib binaries, but I forget to investigate the affected file.
Now I just triggered it during nix build ... of my project. The resulting binary has a normal size, but first 2000 hex bytes seemed to be zeroed:
> hexyl /nix/store/m1sg718mjq5rh8cm8yhdkcj3w671263a-bfte-0.1.0/bin/bfte | head -n 5
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 00 00 00 00 00 00 00 00 ┊ 00 00 00 00 00 00 00 00 │⋄⋄⋄⋄⋄⋄⋄⋄┊⋄⋄⋄⋄⋄⋄⋄⋄│
│* │ ┊ │ ┊ │
│00002000│ 64 5f 47 65 74 49 50 49 ┊ 6e 66 6f 00 5f 55 6e 77 │d_GetIPI┊nfo⋄_Unw│
│00002010│ 69 6e 64 5f 47 65 74 52 ┊ 65 67 69 6f 6e 53 74 61 │ind_GetR┊egionSta│
I'm guessing Unsupported version 0 of Verneed record is the symptom of the same corruption, but I'll try to confirm the next time I spot it.
I am not even really sure if this is caused bcachefs, it's just a suspicion. Other than this corruption happening only during compilation of larger Rust projects, my system perfectly fine.
# but this has been happening with many versions of kernels as I'm updating quite frequently
> uname -a
Linux ren 6.17.0-rc2 #1-NixOS SMP PREEMPT_DYNAMIC Tue Jan 1 00:00:00 UTC 1980 x86_64 GNU/Linux
> sudo bcachefs show-super /dev/nvme1n1p2
Device: (unknown device)
External UUID: a933c02c-19d2-40d7-b5d7-42892bd5e154
Internal UUID: 61d26938-b11f-42f0-8968-372a21e8b739
Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef
Device index: 1
Label: (none)
Version: 1.28: inode_has_case_insensitive
Incompatible features allowed: 0.0: (unknown version)
Incompatible features in use: 0.0: (unknown version)
Version upgrade complete: 1.28: inode_has_case_insensitive
Oldest version on disk: 1.3: rebalance_work
Created: Sun Jan 28 21:07:10 2024
Sequence number: 718
Time of last write: Fri Sep 5 12:27:22 2025
Superblock size: 6.20 KiB/1.00 MiB
Clean: 0
Devices: 2
Sections: members_v1,crypt,replicas_v0,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade,recovery_passes
Features: zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done
Options:
block_size: 512 B
btree_node_size: 256 KiB
errors: continue [fix_safe] panic ro
write_error_timeout: 30
metadata_replicas: 2
data_replicas: 1
metadata_replicas_required: 1
data_replicas_required: 1
encoded_extent_max: 64.0 KiB
metadata_checksum: none [crc32c] crc64 xxhash
data_checksum: none [crc32c] crc64 xxhash
checksum_err_retry_nr: 3
compression: zstd
background_compression: none
str_hash: crc32c crc64 [siphash]
metadata_target: none
foreground_target: none
background_target: none
promote_target: none
erasure_code: 0
casefold: 0
inodes_32bit: 1
shard_inode_numbers_bits: 5
inodes_use_key_cache: 1
gc_reserve_percent: 8
gc_reserve_bytes: 0 B
root_reserve_percent: 0
wide_macs: 0
promote_whole_extents: 0
acl: 1
usrquota: 0
grpquota: 0
prjquota: 0
degraded: [ask] yes very no
journal_flush_delay: 1000
journal_flush_disabled: 0
journal_reclaim_delay: 100
journal_transaction_names: 1
allocator_stuck_timeout: 30
version_upgrade: [compatible] incompatible none
nocow: 0
rebalance_on_ac_only: 0
members_v2 (size 448):
Device: 1
Label: (none)
UUID: 4bd08f3b-030e-4cd1-8b1e-1f3c8662b455
Size: 3.72 TiB
read errors: 32
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 1.00 MiB
First bucket: 0
Buckets: 3906505
Last mount: Fri Sep 5 12:27:22 2025
Last superblock write: 718
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,user
Btree allocated bitmap blocksize: 32.0 MiB
Btree allocated bitmap: 0000010000000000000000000000000000000000000000100000000000101111
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
Device: 2
Label: (none)
UUID: 40b54415-da88-4eb9-a8fa-33095dfe763b
Size: 3.72 TiB
read errors: 0
write errors: 0
checksum errors: 0
seqread iops: 0
seqwrite iops: 0
randread iops: 0
randwrite iops: 0
Bucket size: 2.00 MiB
First bucket: 0
Buckets: 1953253
Last mount: Fri Sep 5 12:27:22 2025
Last superblock write: 718
State: rw
Data allowed: journal,btree,user
Has data: journal,btree,user
Btree allocated bitmap blocksize: 64.0 MiB
Btree allocated bitmap: 0000000000000000000000000100000000000000000000100000000000000111
Durability: 1
Discard: 1
Freespace initialized: 1
Resize on mount: 0
errors (size 200):
btree_node_bset_older_than_sb_min 1 Sat Apr 27 17:18:02 2024
fs_usage_data_wrong 1 Sat Apr 27 17:20:43 2024
fs_usage_replicas_wrong 1 Sat Apr 27 17:20:48 2024
dev_usage_sectors_wrong 1 Sat Apr 27 17:20:36 2024
dev_usage_fragmented_wrong 1 Sat Apr 27 17:20:39 2024
alloc_key_dirty_sectors_wrong 3 Sat Apr 27 17:20:35 2024
bucket_sector_count_overflow 1 Sat Apr 27 16:42:51 2024
backpointer_to_missing_ptr 5 Sat Apr 27 17:21:53 2024
ptr_to_missing_backpointer 2 Sat Apr 27 17:21:57 2024
key_in_missing_inode 5 Sat Apr 27 17:22:48 2024
accounting_key_version_0 8 Fri Oct 25 19:00:01 2024
inode_dir_has_nonzero_i_size 491072 Tue Jul 1 16:30:49 2025
Hmmm... Actually I just got:
> ./target/debug/bfte
./target/debug/bfte: error while loading shared libraries: ./target/debug/bfte: unsupported version 0 of Verneed record
and at least the start looks OK(-ish?):
> hexyl target/debug/bfte | head -n 5
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 7f 45 4c 46 02 01 01 00 ┊ 00 00 00 00 00 00 00 00 │•ELF•••⋄┊⋄⋄⋄⋄⋄⋄⋄⋄│
│00000010│ 03 00 3e 00 01 00 00 00 ┊ 40 42 b9 00 00 00 00 00 │•⋄>⋄•⋄⋄⋄┊@B×⋄⋄⋄⋄⋄│
│00000020│ 40 00 00 00 00 00 00 00 ┊ 28 0d a9 0a 00 00 00 00 │@⋄⋄⋄⋄⋄⋄⋄┊(_×_⋄⋄⋄⋄│
│00000030│ 00 00 00 00 40 00 38 00 ┊ 0d 00 40 00 34 00 31 00 │⋄⋄⋄⋄@⋄8⋄┊_⋄@⋄4⋄1⋄│
> readelf -h ./target/debug/bfte
ELF Header:
Magic: 7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
Class: ELF64
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Position-Independent Executable file)
Machine: Advanced Micro Devices X86-64
Version: 0x1
Entry point address: 0xb94240
Start of program headers: 64 (bytes into file)
Start of section headers: 178851112 (bytes into file)
Flags: 0x0
Size of this header: 64 (bytes)
Size of program headers: 56 (bytes)
Number of program headers: 13
Size of section headers: 64 (bytes)
Number of section headers: 52
Section header string table index: 49
so it's not that simple.
I have the same Nix expression which consistently produces the zerod first-8K binary on my desktop system with bcachefs, while it produces a working binary on my laptop. These machines have different hardware, but the configuration is using NixOS so they should be very close to each other in software otherwise.
After a lot more differential testing (disabling zstd, zram swap, using different build options, symlinking target dir tmpfs), I suspect this is not related to bcachefs after all, so closing. Sorry for the noise and will reopen if I actually do have some more evidence this is bcachefs fault.
There was another bug report that pointed at a possible bcachefs issue (it was subvolumes related, oddly) that this sounded similar to:
https://github.com/rust-lang/rust/issues/143428
I was never able to reproduce it, so let me know if you find anything new that points to bcachefs.
Huh. It does indeed look very much like my issue.
I've discovered that the zero-header issue disappears if I disable Determinate Nix and switch to vanilla Nix on my system and reported it: https://github.com/DeterminateSystems/nix-src/issues/204 , but I think it might be causing it only indirectly, or it could be just a changed timing of something somewhere.
What's weird is that I believe Nix is building derivations under /tmp/<somedir> and my tmp is a tmpfs, so it should not even be touching bcachefs.
My project is also using mold for linking:
> export | grep mold
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS '-C link-arg=-fuse-ld=mold -C link-arg=-Wl,--compress-debug-…
I'll post if I ever find any updates, but I'm afraid this is going to be hard to get to the bottom of.
So, if I can get this to reproduce in ktest it should be fairly easy to track down - that was the roadblock before.
Is deterministic nix something that could be installed via the nix package manager on a debian install?
or possibly just a chroot, that's how I was trying to reproduce the report on Arch
Determinate Nix is can be installed on any Linux using Determinate Nix Installer. It has an option/question which Nix to install, I think. Just to be clear: Determinate Nix Installer can install either Standard or Determinate Nix.
I'm not aware of any way to install it in a chroot/container. But it has an uninstaller that should remove it quite cleanly, so no litter should be left on your system if you want to get rid of it afterwards.
I'm under strong impression that the issue is related to memory pressure. The more memory (swap?) is used, the more likely it is to be triggered.
We finally had a user in the IRC channel stumble across this one, and fairly quickly tracked it down to a pagecache inconsistency...
which then led to much swearing over what's been going on with large folios, and that took a bit to figure out how to deal with... subject for another bug...
but it's finally fixed in master, and will be in the next release