bcachefs
bcachefs copied to clipboard
Published
20 hours ago •
koverstreet
koverstreet
Scheduling while atomic, followed by deadlock [95ff72a6]
bcachefs fs usage:
Filesystem: a4b165a2-556b-4650-88b8-d90f8ee4b473
Size: 13352768310272
Used: 2831773152768
Online reserved: 26353664
Data type Required/total Devices
reserved: 1/1 [] 1318608896
btree: 1/2 [sda2 sdb2] 12582912
btree: 1/2 [sda2 nvme0n1] 59768832
btree: 1/2 [sda2 sdd] 15728640
btree: 1/2 [sdb2 sdd] 13107200
btree: 1/2 [nvme1n1 nvme0n1] 28808052736
btree: 1/2 [sda2 sdc] 11010048
btree: 1/2 [sda2 nvme1n1] 553123840
btree: 1/2 [sdb2 sdc] 17301504
btree: 1/2 [sdb2 nvme1n1] 524288
btree: 1/2 [sdc sdd] 39845888
user: 1/2 [sda2 sdc] 503077970432
user: 1/2 [sdc sdd] 599718802944
user: 1/2 [sdb2 sdc] 397514276864
user: 1/2 [nvme1n1 nvme0n1] 1224558592
user: 1/2 [sda2 sdb2] 374970234368
user: 1/2 [sda2 sdd] 509600132096
user: 1/2 [sdb2 sdd] 393567947264
cached: 1/1 [sdc] 143165249536
cached: 1/1 [sda2] 78861477888
cached: 1/1 [nvme1n1] 121419318272
cached: 1/1 [sdb2] 107098314240
cached: 1/1 [sdd] 143159793152
cached: 1/1 [nvme0n1] 119172525568
rotational (device 0): sda2 rw
data buckets fragmented
free: 0 3958408
sb: 3149824 7 520192
journal: 4294967296 8192
btree: 326107136 1177 290979840
user: 693744751104 1395455 37881076736
cached: 78861477888 358983
parity: 0 0
stripe: 0 0
need_gc_gens: 0 0
need_discard: 0 0
erasure coded: 0 0
capacity: 3000092327936 5722222
rotational (device 1): sdb2 rw
data buckets fragmented
free: 0 4232502
sb: 3149824 7 520192
journal: 4294967296 8192
btree: 21757952 47 2883584
user: 583051507712 1126967 7812890112
cached: 107098314240 354507
parity: 0 0
stripe: 0 0
need_gc_gens: 0 0
need_discard: 0 0
erasure coded: 0 0
capacity: 3000092327936 5722222
rotational (device 2): sdc rw
data buckets fragmented
free: 0 5706216
sb: 3149824 7 520192
journal: 4294967296 8192
btree: 34078720 81 8388608
user: 750182230016 1451732 10953983488
cached: 143165249536 464667
parity: 0 0
stripe: 0 0
need_gc_gens: 0 0
need_discard: 0 0
erasure coded: 0 0
capacity: 4000786677760 7630895
rotational (device 3): sdd rw
data buckets fragmented
free: 0 5705650
sb: 3149824 7 520192
journal: 4294967296 8192
btree: 34340864 82 8650752
user: 751470875136 1453564 10625408512
cached: 143159793152 463400
parity: 0 0
stripe: 0 0
need_gc_gens: 0 0
need_discard: 0 0
erasure coded: 0 0
capacity: 4000786677760 7630895
ssd (device 5): nvme0n1 rw
data buckets fragmented
free: 0 210834
sb: 3149824 7 520192
journal: 2000158720 3815
btree: 14433910784 40925 7022575616
user: 612279296 1232 33643520
cached: 119172525568 231574
parity: 0 0
stripe: 0 0
need_gc_gens: 0 0
need_discard: 0 9
erasure coded: 0 0
capacity: 256060162048 488396
ssd (device 4): nvme1n1 rw
data buckets fragmented
free: 0 441857
sb: 3149824 13 258048
journal: 2000420864 7631
btree: 14680850432 56003
user: 612279296 2406 18439168
cached: 121419318272 468874
parity: 0 0
stripe: 0 0
need_gc_gens: 0 0
need_discard: 0 9
erasure coded: 0 0
capacity: 256060424192 976793
bcachefs show-super:
ubuntu@ubuntu:~$ sudo bcachefs show-super /dev/nvme0n1
External UUID: a4b165a2-556b-4650-88b8-d90f8ee4b473
Internal UUID: 7eefb620-ddab-4859-9449-9ca659257331
Device index: 5
Label:
Version: backpointers
Oldest version on disk: subvol_dirent
Created: Tue Nov 2 17:56:16 2021
Sequence number: 305
Superblock size: 37496
Clean: 0
Devices: 6
Sections: journal,members,crypt,replicas_v0,disk_groups,clean,journal_seq_blacklist,counters
Features: lz4,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,incompressible,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_fl
Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done
Options:
block_size: 4.00 KiB
btree_node_size: 256 KiB
errors: continue [ro] panic
metadata_replicas: 2
data_replicas: 2
metadata_replicas_required: 1
data_replicas_required: 1
encoded_extent_max: 64.0 KiB
metadata_checksum: none [crc32c] crc64 xxhash
data_checksum: none [crc32c] crc64 xxhash
compression: none [lz4] gzip zstd
background_compression: [none] lz4 gzip zstd
str_hash: crc32c crc64 [siphash]
metadata_target: ssd
foreground_target: ssd
background_target: rotational
promote_target: ssd
erasure_code: 0
inodes_32bit: 1
shard_inode_numbers: 1
inodes_use_key_cache: 1
gc_reserve_percent: 8
gc_reserve_bytes: 0 B
root_reserve_percent: 0
wide_macs: 0
acl: 1
usrquota: 0
grpquota: 0
prjquota: 0
journal_flush_delay: 1000
journal_flush_disabled: 0
journal_reclaim_delay: 1000
members (size 344):
Device: 0
UUID: 8f6bbc2d-3b2b-4423-ad13-1a50108279e0
Size: 2.73 TiB
Bucket size: 512 KiB
First bucket: 0
Buckets: 5722222
Last mount: Fri Jul 1 13:04:42 2022
State: rw
Label: rotational (0)
Data allowed: journal,btree,user
Has data: journal,btree,user,cached
Discard: 0
Freespace initialized: 1
Device: 1
UUID: 93b4098e-bba5-4577-8576-c2cd25086a6d
Size: 2.73 TiB
Bucket size: 512 KiB
First bucket: 0
Buckets: 5722222
Last mount: Fri Jul 1 13:04:42 2022
State: rw
Label: rotational (0)
Data allowed: journal,btree,user
Has data: journal,btree,user,cached
Discard: 0
Freespace initialized: 1
Device: 2
UUID: 67701b0f-e143-4d48-8280-12dd5999b168
Size: 3.64 TiB
Bucket size: 512 KiB
First bucket: 0
Buckets: 7630895
Last mount: Fri Jul 1 13:04:42 2022
State: rw
Label: rotational (0)
Data allowed: journal,btree,user
Has data: journal,btree,user,cached
Discard: 0
Freespace initialized: 1
Device: 3
UUID: f7129d1d-25a9-4de4-9d16-e9fbd03d0629
Size: 3.64 TiB
Bucket size: 512 KiB
First bucket: 0
Buckets: 7630895
Last mount: Fri Jul 1 13:04:42 2022
State: rw
Label: rotational (0)
Data allowed: journal,btree,user
Has data: journal,btree,user,cached
Discard: 0
Freespace initialized: 1
Device: 4
UUID: 1f0dfbdb-b149-4cc9-9b3e-f3624bee4b43
Size: 238 GiB
Bucket size: 256 KiB
First bucket: 0
Buckets: 976793
Last mount: Fri Jul 1 13:04:42 2022
State: rw
Label: ssd (1)
Data allowed: journal,btree,user
Has data: journal,btree,user,cached
Discard: 1
Freespace initialized: 1
Device: 5
UUID: 0ceeb0bd-0b50-4b96-ab96-305c8380b3e9
Size: 238 GiB
Bucket size: 512 KiB
First bucket: 0
Buckets: 488396
Last mount: Fri Jul 1 13:04:42 2022
State: rw
Label: ssd (1)
Data allowed: journal,btree,user
Has data: journal,btree,user,cached
Discard: 1
Freespace initialized: 1
dmesg:
[73569.424497] bcachefs: bch2_fs_open() ret 0
[73601.041795] kmemleak: 74 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
[74229.571709] kmemleak: 6 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
[74301.921545] perf: interrupt took too long (3170 > 3127), lowering kernel.perf_event_max_sample_rate to 63000
[74852.140736] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
[76044.583016] BUG: scheduling while atomic: kworker/u16:3/11114/0x00000102
[76044.583026] Modules linked in: nfnetlink snd_seq_dummy snd_hrtimer zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) intel_rapl_msr mei_hdcp mei_pxp intel_rapl_common intel_tcc_cooling x86_pkg_temp_th
[76044.583148] crct10dif_pclmul drm_kms_helper crc32_pclmul ghash_clmulni_intel aesni_intel syscopyarea crypto_simd sysfillrect sysimgblt fb_sys_fops nvme cryptd psmouse e1000e drm ahci nvme_core i2c_i801 i2c_smbus xhci_pci libahci xhci_p
[76044.583178] CPU: 3 PID: 11114 Comm: kworker/u16:3 Tainted: P O 5.18.7+bcachefs.git20220628.95ff72a6-1-debug #1-Ubuntu
[76044.583182] Hardware name: System manufacturer System Product Name/Z170 PRO GAMING, BIOS 1904 07/05/2016
[76044.583184] Workqueue: events_unbound __bch2_read_endio
[76044.583190] Call Trace:
[76044.583191] <IRQ>
[76044.583193] show_stack+0x56/0x5d
[76044.583198] dump_stack_lvl+0x4d/0x64
[76044.583201] dump_stack+0x14/0x17
[76044.583203] __schedule_bug.cold+0xb7/0xc8
[76044.583207] __schedule+0xb61/0x10c0
[76044.583210] ? osq_lock+0x254/0x580
[76044.583214] ? io_schedule_timeout+0x160/0x160
[76044.583217] ? kasan_check_range+0x182/0x1e0
[76044.583221] schedule+0xee/0x210
[76044.583224] schedule_preempt_disabled+0x19/0x30
[76044.583226] __mutex_lock.constprop.0+0xbe6/0xd80
[76044.583229] ? secondary_startup_64_no_verify+0xd5/0xdb
[76044.583233] ? kernel_text_address+0x1c/0xe0
[76044.583236] ? create_prof_cpu_mask+0x30/0x30
[76044.583239] ? ww_mutex_lock+0x1c0/0x1c0
[76044.583242] ? unwind_next_frame+0x3b/0x50
[76044.583245] ? arch_stack_walk+0x8c/0xf0
[76044.583248] ? secondary_startup_64_no_verify+0xd5/0xdb
[76044.583251] __mutex_lock_slowpath+0x13/0x20
[76044.583254] mutex_lock+0xd4/0xe0
[76044.583256] ? __mutex_lock_slowpath+0x20/0x20
[76044.583259] ? __stack_depot_save+0x3a/0x5a0
[76044.583262] writepoint_find+0x102/0x810
[76044.583266] ? kasan_record_aux_stack_noalloc+0xf/0x20
[76044.583269] ? call_rcu+0x76/0x7b0
[76044.583271] ? put_object+0x6b/0x90
[76044.583274] ? kmem_cache_free+0x12c/0x350
[76044.583276] ? mempool_free_slab+0x1b/0x20
[76044.583279] ? mempool_free+0xde/0x3a0
[76044.583282] ? bio_free+0xd9/0x120
[76044.583285] ? bio_put+0x188/0x310
[76044.583288] ? bch2_readpages_end_io+0x57/0x440
[76044.583291] ? bio_endio+0x417/0x580
[76044.583293] bch2_alloc_sectors_start+0x17b/0xf90
[76044.583296] ? blk_complete_reqs+0xb1/0xf0
[76044.583300] ? blk_done_softirq+0x21/0x30
[76044.583303] ? do_idle+0xd7/0x190
[76044.583305] ? secondary_startup_64_no_verify+0xd5/0xdb
[76044.583308] ? get_stack_info_noinstr+0x1d/0x120
[76044.583312] ? bch2_writepoint_stop+0x50/0x50
[76044.583314] ? gen_poly_key.isra.0+0x640/0x640
[76044.583317] ? get_stack_info_noinstr+0x1d/0x120
[76044.583320] __bch2_write+0x3fd/0x1310
[76044.583323] ? bch2_checksum_bio+0x160/0x160
[76044.583326] ? unwind_next_frame+0x3b/0x50
[76044.583328] ? arch_stack_walk+0x8c/0xf0
[76044.583331] ? bch2_submit_wbio_replicas+0xd60/0xd60
[76044.583333] ? sched_clock+0xd/0x10
[76044.583336] ? sched_clock_cpu+0x1c/0x1a0
[76044.583338] ? ret_from_fork+0x22/0x30
[76044.583341] bch2_write+0x6ad/0x9f0
[76044.583344] bch2_data_update_read_done+0x2ea/0x3b0
[76044.583347] ? bch2_data_update_index_update+0x2860/0x2860
[76044.583350] ? bio_copy_data_iter+0x6ac/0xfa0
[76044.583353] ? kmem_cache_free+0x12c/0x350
[76044.583355] ? mempool_free_slab+0x1b/0x20
[76044.583358] ? mempool_free+0xde/0x3a0
[76044.583361] ? sg_pool_free+0x49/0x80
[76044.583363] promote_start+0x294/0x480
[76044.583367] __bch2_read_endio+0x13d3/0x18a0
[76044.583369] ? bio_alloc_bioset+0x356/0x480
[76044.583372] ? bch2_write_bio_alloc.isra.0+0x17b/0x370
[76044.583374] ? bch2_write_extent+0x270/0x20a0
[76044.583376] ? __bch2_write+0x6aa/0x1310
[76044.583378] ? bch2_write+0x6ad/0x9f0
[76044.583380] ? bch2_data_update_read_done+0x2ea/0x3b0
[76044.583384] ? bch2_rbio_error+0x450/0x450
[76044.583386] ? __raw_spin_lock_irqsave+0x8e/0xf0
[76044.583389] ? wake_up_process+0x19/0x20
[76044.583392] ? bch2_latency_acct+0x134/0x180
[76044.583395] bch2_read_endio+0x3f4/0xb80
[76044.583397] ? bch2_read_endio+0x3f4/0xb80
[76044.583399] bio_endio+0x417/0x580
[76044.583403] blk_update_request+0x2c0/0xf30
[76044.583406] scsi_end_request+0x77/0x600
[76044.583409] scsi_io_completion+0x142/0x5a0
[76044.583412] ? scsi_run_host_queues+0x70/0x70
[76044.583414] ? __kasan_check_write+0x18/0x20
[76044.583417] ? scsi_device_unbusy+0x176/0x390
[76044.583419] scsi_finish_command+0x280/0x4a0
[76044.583423] scsi_complete+0x10a/0x350
[76044.583425] blk_complete_reqs+0xb1/0xf0
[76044.583428] blk_done_softirq+0x21/0x30
[76044.583431] __do_softirq+0x194/0x5f7
[76044.583435] __irq_exit_rcu+0xe7/0x130
[76044.583437] irq_exit_rcu+0x12/0x20
[76044.583440] common_interrupt+0x8c/0xa0
[76044.583442] </IRQ>
[76044.583443] <TASK>
[76044.583444] asm_common_interrupt+0x22/0x40
[76044.583447] RIP: 0010:chacha_8block_xor_avx2+0x271/0x7c0
[76044.583450] Code: e8 c4 62 15 00 eb c5 dd fe 44 24 60 c5 fd 7f 44 24 60 c5 0d ef f0 c4 62 0d 00 f3 c4 41 2d fe d7 c4 c1 55 ef ea c5 fd 72 f5 0c <c5> d5 72 d5 14 c5 d5 eb e8 c4 41 25 fe dc c4 c1 4d ef f3 c5 fd 72
[76044.583454] RSP: 0018:ffffc9001c25ed80 EFLAGS: 00000206
[76044.583457] RAX: 0000000000000a00 RBX: 0000000000000800 RCX: 0000000000000a00
[76044.583459] RDX: ffff8881c527a600 RSI: ffff8881c527a600 RDI: ffffc9001c25efa0
[76044.583461] RBP: ffffc9001c25ee70 R08: 000000000000000c R09: 0000000000000000
[76044.583463] R10: ffffc9001c25ee20 R11: 1ffff9200384bde0 R12: ffff8881c527a800
[76044.583465] R13: ffff8881c527a800 R14: ffffc9001c25efa0 R15: 0000000000000014
[76044.583469] ? save_fpregs_to_fpstate+0x140/0x140
[76044.583472] ? chacha_dosimd+0x185/0x5d0
[76044.583475] chacha_simd_stream_xor+0x504/0x680
[76044.583478] ? update_stack_state+0x2d4/0x600
[76044.583481] ? chacha_init_arch+0x810/0x810
[76044.583484] ? update_stack_state+0x2d4/0x600
[76044.583487] ? update_stack_state+0x2d4/0x600
[76044.583490] chacha_simd+0x63/0x80
[76044.583493] crypto_skcipher_encrypt+0xf1/0x160
[76044.583497] do_encrypt_sg+0xaa/0x110
[76044.583499] ? bch2_chardev_exit+0x80/0x80
[76044.583502] ? ret_from_fork+0x22/0x30
[76044.583505] ? stack_trace_save+0x98/0xd0
[76044.583508] ? __stack_depot_save+0x3a/0x5a0
[76044.583510] ? kasan_save_stack+0x3e/0x50
[76044.583513] ? bch2_checksum_merge+0x24e/0x3e0
[76044.583516] ? bch2_encrypt_bio+0x970/0x970
[76044.583519] ? __kasan_check_read+0x15/0x20
[76044.583522] ? rmqueue+0x692/0x1b20
[76044.583525] ? kernel_text_address+0x1c/0xe0
[76044.583528] ? memset+0x40/0x50
[76044.583531] bch2_encrypt_bio+0x58c/0x970
[76044.583534] ? bch2_rechecksum_bio+0x6af/0x1330
[76044.583537] ? bch2_checksum_bio+0x160/0x160
[76044.583543] ? bch2_write_extent+0x886/0x20a0
[76044.583545] ? bch2_write_extent+0x886/0x20a0
[76044.583548] bch2_write_extent+0x9c4/0x20a0
[76044.583551] ? bch2_write_bio_alloc.isra.0+0x370/0x370
[76044.583554] ? bch2_writepoint_stop+0x50/0x50
[76044.583557] ? bch2_encrypt_bio+0x58c/0x970
[76044.583560] __bch2_write+0x6aa/0x1310
[76044.583562] ? __bch2_write+0x6aa/0x1310
[76044.583565] ? bch2_submit_wbio_replicas+0xd60/0xd60
[76044.583568] bch2_write+0x6ad/0x9f0
[76044.583571] bch2_data_update_read_done+0x2ea/0x3b0
[76044.583574] ? bch2_data_update_index_update+0x2860/0x2860
[76044.583578] promote_start+0x294/0x480
[76044.583581] __bch2_read_endio+0x13d3/0x18a0
[76044.583584] ? psi_group_change+0x857/0xd70
[76044.583587] ? update_load_avg+0x1c3/0x1b60
[76044.583590] ? bch2_rbio_error+0x450/0x450
[76044.583592] ? psi_task_switch+0x1a6/0x540
[76044.583596] ? raw_spin_rq_unlock+0x1b/0x90
[76044.583599] ? __schedule+0x632/0x10c0
[76044.583602] process_one_work+0x77b/0x11c0
[76044.583605] ? bch2_rbio_error+0x450/0x450
[76044.583607] ? process_one_work+0x77b/0x11c0
[76044.583609] ? _raw_spin_lock_irq+0x8c/0xe0
[76044.583612] worker_thread+0x548/0x1180
[76044.583614] ? io_schedule_timeout+0x160/0x160
[76044.583618] kthread+0x27a/0x320
[76044.583620] ? process_one_work+0x11c0/0x11c0
[76044.583622] ? kthread_complete_and_exit+0x30/0x30
[76044.583624] ret_from_fork+0x22/0x30
[76044.583628] </TASK>```
Hit this with a preempt-debugging kernel (upstream commit 04962c7b):
[ 2201.405869] bcachefs (a4b165a2-556b-4650-88b8-d90f8ee4b473): checking xattrs
[ 2201.414923] bcachefs (a4b165a2-556b-4650-88b8-d90f8ee4b473): checking root directory
[ 2242.858647] bcachefs (a4b165a2-556b-4650-88b8-d90f8ee4b473): checking inode nlinks
[ 2264.016463] bcachefs (a4b165a2-556b-4650-88b8-d90f8ee4b473): fsck done
[ 2264.772563] bcachefs (a4b165a2-556b-4650-88b8-d90f8ee4b473): ret (No error)
[ 2264.773425] bcachefs (a4b165a2-556b-4650-88b8-d90f8ee4b473): mounted version=backpointers opts=metadata_replicas=2,d
[ 2264.773447] bcachefs: bch2_fs_open() ret 0
[ 2674.368127] kmemleak: 4 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
[ 3313.405568] kmemleak: 5 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
[ 5190.350551] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Quota mode: none.
[ 5190.350595] ext4 filesystem being mounted at /boot supports timestamps until 2038 (0x7fffffff)
[ 5190.391124] FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
[ 5324.854465] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:280
[ 5324.854474] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 0, name: swapper/6
[ 5324.854478] preempt_count: 101, expected: 0
[ 5324.854481] RCU nest depth: 0, expected: 0
[ 5324.854484] Preemption disabled at:
[ 5324.854486] [<ffffffffa27671c3>] __raw_spin_lock_irqsave+0x23/0x40
[ 5324.854497] CPU: 6 PID: 0 Comm: swapper/6 Tainted: P O 5.18.14+bcachefs.git20220725.04962c7b-1-debug
[ 5324.854502] Hardware name: System manufacturer System Product Name/Z170 PRO GAMING, BIOS 1904 07/05/2016
[ 5324.854504] Call Trace:
[ 5324.854507] <IRQ>
[ 5324.854509] show_stack+0x52/0x5b
[ 5324.854515] dump_stack_lvl+0x4e/0x69
[ 5324.854519] ? __raw_spin_lock_irqsave+0x23/0x40
[ 5324.854523] dump_stack+0x14/0x1a
[ 5324.854526] __might_resched.cold+0x20c/0x258
[ 5324.854532] __might_sleep+0xb8/0x170
[ 5324.854537] mutex_lock+0x7c/0xf0
[ 5324.854541] ? __mutex_lock_slowpath+0x20/0x20
[ 5324.854546] writepoint_find+0x122/0x8b0
[ 5324.854552] ? update_stack_state+0x29b/0x660
[ 5324.854556] ? nvme_irq+0xd8/0xf0 [nvme]
[ 5324.854564] bch2_alloc_sectors_start+0x18a/0xe00
[ 5324.854568] ? unwind_next_frame+0x189/0x8d0
[ 5324.854573] ? get_stack_info_noinstr+0x1d/0x120
[ 5324.854577] ? bch2_writepoint_stop+0x50/0x50
[ 5324.854581] ? gen_poly_key.isra.0+0x660/0x660
[ 5324.854585] ? update_stack_state+0x29b/0x660
[ 5324.854589] ? get_stack_info_noinstr+0x1d/0x120
[ 5324.854594] __bch2_write+0x3f0/0x13d0
[ 5324.854598] ? bch2_checksum_bio+0x160/0x160
[ 5324.854604] ? arch_stack_walk+0x8c/0x100
[ 5324.854609] ? bch2_submit_wbio_replicas+0xe20/0xe20
[ 5324.854612] ? sched_clock+0xd/0x20
[ 5324.854616] ? sched_clock_cpu+0x1c/0x1f0
[ 5324.854621] bch2_write+0x69c/0x9f0
[ 5324.854625] ? filter_irq_stacks+0xa0/0xa0
[ 5324.854630] bch2_data_update_read_done+0x2df/0x3a0
[ 5324.854634] ? bch2_data_update_index_update+0x2940/0x2940
[ 5324.854638] ? kmem_cache_free+0x12a/0x360
[ 5324.854644] ? bio_copy_data_iter+0x738/0xe50
[ 5324.854648] ? mempool_free+0xd2/0x3c0
[ 5324.854653] ? sg_pool_free+0x49/0x80
[ 5324.854657] promote_start+0x29a/0x490
[ 5324.854662] __bch2_read_endio+0x1357/0x1810
[ 5324.854666] ? _raw_spin_unlock_irqrestore+0x2b/0x50
[ 5324.854670] ? try_to_wake_up+0x537/0x13b0
[ 5324.854675] ? bch2_rbio_error+0x440/0x440
[ 5324.854679] ? __kasan_check_read+0x15/0x20
[ 5324.854683] ? __kasan_check_read+0x15/0x20
[ 5324.854686] ? _raw_spin_unlock_irqrestore+0x2b/0x50
[ 5324.854692] ? bch2_latency_acct+0x130/0x180
[ 5324.854696] bch2_read_endio+0x3f4/0xb90
[ 5324.854699] ? bch2_read_endio+0x3f4/0xb90
[ 5324.854704] bio_endio+0x445/0x5c0
[ 5324.854709] blk_update_request+0x340/0xfb0
[ 5324.854713] ? ata_qc_complete+0x394/0xc30
[ 5324.854719] scsi_end_request+0x77/0x600
[ 5324.854723] scsi_io_completion+0xf4/0xd60
[ 5324.854727] ? scsi_run_host_queues+0x70/0x70
[ 5324.854730] ? __kasan_check_write+0x18/0x20
[ 5324.854733] ? scsi_device_unbusy+0x176/0x390
[ 5324.854737] scsi_finish_command+0x27e/0x4a0
[ 5324.854742] scsi_complete+0x10a/0x350
[ 5324.854746] blk_complete_reqs+0xab/0xf0
[ 5324.854750] blk_done_softirq+0x55/0x80
[ 5324.854753] __do_softirq+0x1cb/0x660
[ 5324.854758] __irq_exit_rcu+0x95/0x1a0
[ 5324.854762] irq_exit_rcu+0x12/0x20
[ 5324.854765] common_interrupt+0xba/0xe0
[ 5324.854769] </IRQ>
[ 5324.854770] <TASK>
[ 5324.854773] asm_common_interrupt+0x2a/0x40
[ 5324.854776] RIP: 0010:cpuidle_enter_state+0x1ac/0xd90
[ 5324.854781] Code: 00 31 ff e8 16 fe fa fd 80 7d c8 00 74 16 9c 58 0f 1f 40 00 f6 c4 02 0f 85 3d 07 00 00 31 ff e8 9a
[ 5324.854787] RSP: 0018:ffffc900001cfd58 EFLAGS: 00000246
[ 5324.854792] RAX: dffffc0000000000 RBX: ffffe8ffffd00d78 RCX: 0000000000000000
[ 5324.854795] RDX: 1ffff110fa9a86a9 RSI: ffffffffa58bea80 RDI: ffff8887d4d43548
[ 5324.854797] RBP: ffffc900001cfda8 R08: 0000000000000000 R09: ffff8887d4d4382b
[ 5324.854800] R10: ffffed10fa9a8705 R11: 071c71c71c71c71c R12: 0000000000000006
[ 5324.854802] R13: ffffffffa6ecfb80 R14: 000004d7ca0eb218 R15: 0000000000000006
[ 5324.854808] ? cpuidle_enter_state+0x18a/0xd90
[ 5324.854811] ? debug_smp_processor_id+0x1b/0x30
[ 5324.854816] cpuidle_enter+0x53/0xb0
[ 5324.854820] cpuidle_idle_call+0x296/0x3b0
[ 5324.854824] ? arch_cpu_idle_exit+0x70/0x70
[ 5324.854828] ? debug_smp_processor_id+0x1b/0x30
[ 5324.854833] do_idle+0xd4/0x190
[ 5324.854837] cpu_startup_entry+0x21/0x30
[ 5324.854840] start_secondary+0x25f/0x300
[ 5324.854845] ? init_freq_invariance+0x140/0x140
[ 5324.854849] ? start_cpu0+0xc/0xc
[ 5324.854854] secondary_startup_64_no_verify+0xd5/0xdb
[ 5324.854861] </TASK>
[ 5325.101402] BUG: scheduling while atomic: swapper/6/0/0x00000102
[ 5325.101440] Modules linked in: nfnetlink snd_seq_dummy snd_hrtimer zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) ic
[ 5325.101588] mfd_aaeon drm_kms_helper crct10dif_pclmul asus_wmi syscopyarea crc32_pclmul sysfillrect sparse_keymap s
[ 5325.101634] Preemption disabled at:
[ 5325.101636] [<ffffffffa27671c3>] __raw_spin_lock_irqsave+0x23/0x40
[ 5325.101647] CPU: 6 PID: 0 Comm: swapper/6 Tainted: P W O 5.18.14+bcachefs.git20220725.04962c7b-1-debug #1-Ubuntu
[ 5325.101652] Hardware name: System manufacturer System Product Name/Z170 PRO GAMING, BIOS 1904 07/05/2016
[ 5325.101655] Call Trace:
[ 5325.101657] <IRQ>
[ 5325.101659] show_stack+0x52/0x5b
[ 5325.101665] dump_stack_lvl+0x4e/0x69
[ 5325.101669] ? __raw_spin_lock_irqsave+0x23/0x40
[ 5325.101673] dump_stack+0x14/0x1a
[ 5325.101676] __schedule_bug.cold+0xf8/0x109
[ 5325.101681] __schedule+0xcd4/0x13a0
[ 5325.101686] ? io_schedule_timeout+0x160/0x160
[ 5325.101690] ? bch2_btree_path_make_mut+0x250/0x250
[ 5325.101696] schedule+0xe9/0x210
[ 5325.101699] bch2_btree_path_traverse+0xb8/0xd0
[ 5325.101703] ? bch2_bucket_alloc_trans+0x1ef/0x8a0
[ 5325.101707] __bch2_btree_iter_peek+0x493/0x2c40
[ 5325.101711] ? __kasan_check_write+0x18/0x20
[ 5325.101715] ? do_raw_spin_lock+0x13c/0x280
[ 5325.101719] ? rwlock_bug.part.0+0x90/0x90
[ 5325.101726] ? btree_trans_peek_key_cache+0x4d0/0x4d0
[ 5325.101730] ? memmove+0x51/0x70
[ 5325.101734] ? get_stack_info_noinstr+0x1d/0x120
[ 5325.101739] ? bch2_path_get+0x6cb/0xed0
[ 5325.101744] bch2_btree_iter_peek_upto+0x415/0x2740
[ 5325.101748] ? bch2_btree_path_set_pos+0x17d0/0x17d0
[ 5325.101754] ? nvme_queue_rq+0x216/0x890 [nvme]
[ 5325.101761] ? bch2_btree_iter_verify_ret.isra.0+0xb00/0xb00
[ 5325.101766] ? bch2_path_get+0xed0/0xed0
[ 5325.101770] ? update_stack_state+0x29b/0x660
[ 5325.101774] ? nvme_queue_rq+0x216/0x890 [nvme]
[ 5325.101779] ? nvme_queue_rq+0x216/0x890 [nvme]
[ 5325.101785] ? bch2_trans_iter_init+0xbc/0x110
[ 5325.101790] bch2_bucket_alloc_trans+0x26a/0x8a0
[ 5325.101794] ? arch_stack_walk+0x8c/0x100
[ 5325.101800] ? bch2_trans_mark_stripe_ptr+0x1130/0x1130
[ 5325.101804] ? bch2_bucket_alloc_trans_early+0xa70/0xa70
[ 5325.101810] ? bch2_bucket_alloc_trans+0x1ef/0x8a0
[ 5325.101814] ? ktime_get+0x64/0x100
[ 5325.101818] ? bch2_bucket_alloc+0x664/0x1100
[ 5325.101822] ? bch2_trans_begin+0x5b6/0x990
[ 5325.101827] bch2_bucket_alloc+0x698/0x1100
[ 5325.101830] ? scsi_complete+0x10a/0x350
[ 5325.101835] ? do_raw_spin_lock+0x13c/0x280