koverstreet
bch-copygc refusing to freeze
On 1ce0cba2b258b62f095b5f94b1397dc4809bbb50 with ACLs, compression, encryption and tiering:
Log
[ 5943.505164] bcachefs (sdb inum 100364 offset 7817736): data checksum error: expected 7153:804d22e741414877 got e451:325e3153d9eac2ce (type 3)
[ 6690.922375] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 16/1512 sectors, 1/2 buckets not moved (move stats: moved 1520 sectors, raced 0 keys, 0 sectors)
...
[19877.891527] PM: suspend entry (deep)
[19892.334779] Filesystems sync: 14.443 seconds
[19892.480127] Freezing user space processes ... (elapsed 0.004 seconds) done.
[19892.484845] OOM killer disabled.
[19892.484849] Freezing remaining freezable tasks ...
[19912.486548] Freezing of tasks failed after 20.001 seconds (1 tasks refusing to freeze, wq_busy=0):
[19912.486634] task:bch-copygc/c97a state:R running task stack: 0 pid: 2331 ppid: 2 flags:0x00004000
[19912.486672] Call Trace:
[19912.486679] ? lock_acquire+0x18d/0x490
[19912.486688] ? bch2_copygc+0x3d5/0x3540
[19912.486695] ? lock_is_held_type+0x98/0x100
[19912.486703] ? find_held_lock+0x2d/0x110
[19912.486710] ? bch2_copygc+0x3d5/0x3540
[19912.486716] ? bch2_copygc+0x3d5/0x3540
[19912.486720] ? lock_acquired+0x367/0xb10
[19912.486730] ? lock_contended+0xbb0/0xbb0
[19912.486741] ? lock_is_held_type+0x98/0x100
[19912.486752] ? bch2_copygc+0xe7c/0x3540
[19912.486778] ? bch2_copygc_wait_amount+0x2b4/0x480
[19912.486787] ? percpu_ref_put_many.constprop.0+0x1c0/0x1c0
[19912.486791] ? lockdep_hardirqs_on_prepare+0x286/0x3f0
[19912.486833] ? rcu_read_lock_sched_held+0x3f/0x70
[19912.486840] ? trace_rcu_grace_period+0x114/0x1a0
[19912.486855] ? bch2_copygc_thread+0x2ad/0x530
[19912.486864] ? bch2_copygc_wait_amount+0x480/0x480
[19912.486868] ? _raw_spin_unlock_irqrestore+0x2d/0x40
[19912.486881] ? __kthread_parkme+0x93/0x110
[19912.486885] ? bch2_copygc_wait_amount+0x480/0x480
[19912.486894] ? kthread+0x340/0x410
[19912.486899] ? _raw_spin_unlock_irq+0x24/0x30
[19912.486903] ? set_kthread_struct+0x100/0x100
[19912.486912] ? ret_from_fork+0x22/0x30
[19912.486952] Restarting kernel threads ... done.
[19912.488698] OOM killer enabled.
[19912.488701] Restarting tasks ... done.
[19913.116645] PM: suspend exit
[19913.116947] PM: suspend entry (s2idle)
[19913.939505] Filesystems sync: 0.822 seconds
[19913.940715] Freezing user space processes ... (elapsed 0.003 seconds) done.
[19913.944536] OOM killer disabled.
[19913.944540] Freezing remaining freezable tasks ...
[19913.944551] ====================================
[19913.944553] WARNING: bch-copygc/c97a/2331 still has locks held!
[19913.944557] 5.13.11 #1-NixOS Tainted: G W
[19913.944560] ------------------------------------
[19913.944562] 1 lock held by bch-copygc/c97a/2331:
[19913.944566] #0: ffff8881ebc43c30 (&c->btree_trans_barrier){....}-{0:0}, at: bch2_trans_init+0x17a/0x520
[19913.944581]
stack backtrace:
[19913.944584] CPU: 3 PID: 2331 Comm: bch-copygc/c97a Tainted: G W 5.13.11 #1-NixOS
[19913.944588] Hardware name: Dell Inc. Latitude E5470/0C8FKJ, BIOS 1.19.3 08/20/2018
[19913.944591] Call Trace:
[19913.944595] dump_stack+0x93/0xc2
[19913.944602] __bch2_move_data+0xb38/0x2c70
[19913.944607] ? mark_lock+0xf6/0x19d0
[19913.944612] ? find_held_lock+0x2d/0x110
[19913.944617] ? bch2_extent_crc_unpack+0x990/0x990
[19913.944625] ? bch2_migrate_write_init+0x1980/0x1980
[19913.944629] ? __lock_acquire+0x165d/0x5620
[19913.944633] ? lock_chain_count+0x20/0x20
[19913.944639] ? sched_clock_cpu+0x18/0x190
[19913.944658] ? sched_clock_cpu+0x18/0x190
[19913.944664] ? __lock_acquire+0x165d/0x5620
[19913.944671] ? lockdep_hardirqs_on_prepare+0x3f0/0x3f0
[19913.944694] ? lock_acquire+0x18d/0x490
[19913.944699] ? __bch2_move_data+0x2bb/0x2c70
[19913.944706] ? __mutex_lock+0xa2e/0xf50
[19913.944710] ? lock_downgrade+0x6d0/0x6d0
[19913.944714] ? bch2_move_data+0x107/0x8f0
[19913.944717] ? mark_lock+0xf6/0x19d0
[19913.944722] ? mutex_lock_io_nested+0xdf0/0xdf0
[19913.944726] ? lock_chain_count+0x20/0x20
[19913.944730] ? __mutex_unlock_slowpath+0xe7/0x610
[19913.944735] ? bch2_extent_crc_unpack+0x990/0x990
[19913.944739] ? bch2_move_data+0x372/0x8f0
[19913.944742] bch2_move_data+0x372/0x8f0
[19913.944750] ? bch_move_stats_init+0x30/0x30
[19913.944754] ? lockdep_hardirqs_on_prepare+0x3f0/0x3f0
[19913.944759] ? lockdep_hardirqs_on_prepare+0x3f0/0x3f0
[19913.944762] ? lockdep_hardirqs_on_prepare+0x3f0/0x3f0
[19913.944771] ? lock_downgrade+0x6d0/0x6d0
[19913.944775] ? bch2_data_job+0xd60/0xd60
[19913.944778] ? u64_swap+0x70/0x70
[19913.944784] bch2_copygc+0x1960/0x3540
[19913.944805] ? percpu_ref_put_many.constprop.0+0x1c0/0x1c0
[19913.944809] ? lockdep_hardirqs_on_prepare+0x286/0x3f0
[19913.944817] ? rcu_read_lock_sched_held+0x3f/0x70
[19913.944824] bch2_copygc_thread+0x2ad/0x530
[19913.944828] ? bch2_copygc_wait_amount+0x480/0x480
[19913.944831] ? _raw_spin_unlock_irqrestore+0x2d/0x40
[19913.944836] ? __kthread_parkme+0x93/0x110
[19913.944840] ? bch2_copygc_wait_amount+0x480/0x480
[19913.944844] kthread+0x340/0x410
[19913.944847] ? _raw_spin_unlock_irq+0x24/0x30
[19913.944851] ? set_kthread_struct+0x100/0x100
[19913.944855] ret_from_fork+0x22/0x30
[19913.945840] (elapsed 0.001 seconds) done.
...
[20355.972958] PM: suspend exit
...
[20433.366582] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 192/4056 sectors, 1/5 buckets not moved (move stats: moved 4616 sectors, raced 0 keys, 0 sectors)
[20460.551139] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 45/4056 sectors, 1/7 buckets not moved (move stats: moved 5368 sectors, raced 0 keys, 0 sectors)
[20570.893980] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 103/4055 sectors, 1/7 buckets not moved (move stats: moved 10888 sectors, raced 0 keys, 0 sectors)
[20658.270565] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 346/10167 sectors, 1/11 buckets not moved (move stats: moved 18560 sectors, raced 0 keys, 0 sectors)
[20674.064093] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 923/33286 sectors, 10/38 buckets not moved (move stats: moved 36776 sectors, raced 0 keys, 0 sectors)
[20681.535741] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 1032/25691 sectors, 7/37 buckets not moved (move stats: moved 24840 sectors, raced 0 keys, 0 sectors)
[20703.577601] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 136/22254 sectors, 2/30 buckets not moved (move stats: moved 34632 sectors, raced 0 keys, 0 sectors)
[20712.451606] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 224/11776 sectors, 2/17 buckets not moved (move stats: moved 12416 sectors, raced 0 keys, 0 sectors)
[20722.791563] bcachefs (c97a9f17-2d9a-49fc-aafd-03f8f9a339c1): copygc finished but 676/49232 sectors, 8/53 buckets not moved (move stats: moved 64200 sectors, raced 0 keys, 0 sectors)
