Deadlock (in bch2_check_for_deadlock?) [3fdcc884]

[RAOF]

As of commit 3fdcc884

After ~12 hours my system ran into what seems to be a deadlock in bcachefs, after which everything touching the fs ends up in the tarpit.

The initial blocked task was:

INFO: task txg_sync:5718 blocked for more than 120 seconds.
      Tainted: P           O       6.1.12+bcachefs.git20230217.3fdcc884-1-generic #1-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task: txg_sync        state:D stack:0     pid:5718  ppid:2      flags:0x00004000
Call Trace:
 <TASK>
 __schedule+0x243/0x5f0
 schedule+0x63/0x110
 __six_lock_type_slowpath.constprop.0+0x25e/0x740
 ? bch2_check_for_deadlock+0x5a0/0x5a0 [bcachefs]
 six_lock_ip_waiter_read+0x13c/0x180
 ? bch2_check_for_deadlock+0x5a0/0x5a0 [bcachefs]
 bch2_btree_node_get+0x7d8/0x810 [bcachefs]
 ? bch2_btree_iter_peek_slot+0x30e/0x6e0 [bcachefs]
 ? bch2_btree_iter_peek_upto+0xb1/0x760 [bcachefs]
 bch2_btree_path_traverse_one+0x265/0x790 [bcachefs]
 ? bch2_btree_iter_peek_upto+0xb1/0x760 [bcachefs]
 __bch2_btree_iter_peek+0xe5/0x5f0 [bcachefs]
 bch2_btree_iter_peek_upto+0xb1/0x760 [bcachefs]
 ? bch2_path_get+0x5e/0x5b0 [bcachefs]
 bch2_bucket_alloc_freelist+0x184/0x380 [bcachefs]
 bch2_bucket_alloc_trans+0x1ec/0x610 [bcachefs]
 bch2_bucket_alloc_set_trans+0x16e/0x430 [bcachefs]
 ? get_buckets_from_writepoint+0x160/0x2a0 [bcachefs]
 open_bucket_add_buckets+0x1e4/0x370 [bcachefs]
 bch2_alloc_sectors_start_trans+0x3ce/0x610 [bcachefs]
 ? sched_clock+0x9/0x10
 __bch2_write+0x1a3/0x790 [bcachefs]
 ? __bch2_write+0x139/0x790 [bcachefs]
 bch2_write+0x1bd/0x270 [bcachefs]
 __bch2_writepage+0x357/0xa60 [bcachefs]
 ? __mod_lruvec_state+0x36/0x50
 write_cache_pages+0x1a2/0x540
 ? bch2_read_single_page.isra.0+0x190/0x190 [bcachefs]
 bch2_writepages+0x83/0x100 [bcachefs]
 do_writepages+0xcf/0x1e0
 ? slab_post_alloc_hook+0x18b/0x220
 filemap_fdatawrite_wbc+0x75/0xb0
 __filemap_fdatawrite_range+0x58/0x80
 file_write_and_wait_range+0x5b/0xc0
 bch2_fsync+0x24/0xa0 [bcachefs]
 vfs_fsync+0x4b/0x90
 ? list_head+0xd/0x40 [zfs]
 zfs_file_fsync+0x14/0x20 [zfs]
 vdev_file_io_start+0x1c4/0x1d0 [zfs]
 zio_vdev_io_start+0x10e/0x2d0 [zfs]
 zio_nowait+0xa6/0x140 [zfs]
 zio_flush+0x30/0x50 [zfs]
 vdev_config_sync+0xc9/0x290 [zfs]
 spa_sync_rewrite_vdev_config+0x83/0x210 [zfs]
 spa_sync+0x2f4/0x5e0 [zfs]
 txg_sync_thread+0x20c/0x290 [zfs]
 ? txg_dispatch_callbacks+0x110/0x110 [zfs]
 ? __thread_exit+0x20/0x20 [spl]
 thread_generic_wrapper+0x5c/0x70 [spl]
 kthread+0xe6/0x110
 ? kthread_complete_and_exit+0x20/0x20
 ret_from_fork+0x1f/0x30
 </TASK>

Here's the full netconsole dmesg including task state dump