linux icon indicating copy to clipboard operation
linux copied to clipboard

System freeze with 5.13.1-rt1-xanmod1

Open cj0nes opened this issue 2 years ago • 9 comments

Hello,

I recently updated my kernel from 5.11.12-rt11-xanmod1 to 5.13.1-rt1-xanmod1 and started experiencing freezing. I fell back to the previous version to regain stability. I hope I've captured the right logging entries, as kernel troubleshooting is quite far outside of my wheelhouse.

Jul 11 18:26:44 pop-os kernel: [  216.337877] ------------[ cut here ]------------
Jul 11 18:26:44 pop-os kernel: [  216.337878] WARNING: CPU: 6 PID: 374 at kfence_handle_page_fault+0xc0/0x220
Jul 11 18:26:44 pop-os kernel: [  216.337883] Modules linked in: snd_seq_dummy snd_hrtimer uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_usb_audio videodev intel_rapl_msr snd_usbmidi_lib mc intel_rapl_common nls_iso8859_1 edac_mce_amd input_leds joydev snd_hda_codec_realtek xpad kvm snd_hda_codec_generic ff_memless ledtrig_audio snd_hda_codec_hdmi rapl snd_hda_intel snd_intel_dspcfg snd_seq_midi snd_seq_midi_event snd_intel_sdw_acpi wmi_bmof snd_rawmidi snd_hda_codec snd_hda_core snd_seq snd_hwdep snd_seq_device snd_pcm ccp snd_timer k10temp snd soundcore mac_hid sch_fq_codel msr parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs dm_crypt raid10 raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx raid6_pq xor libcrc32c raid1 raid0 multipath linear system76_acpi hid_generic usbhid hid amdgpu iommu_v2 gpu_sched drm_ttm_helper ttm drm_kms_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cec aesni_intel rc_core sysimgblt igb syscopyarea crypto_simd ahci
Jul 11 18:26:44 pop-os kernel: [  216.337908]  sysfillrect fb_sys_fops xhci_pci dca cryptd libahci i2c_piix4 xhci_pci_renesas i2c_algo_bit drm nvme nvme_core wmi
Jul 11 18:26:44 pop-os kernel: [  216.337912] CPU: 6 PID: 374 Comm: kworker/u64:7 Tainted: G        W         5.13.1-rt1-xanmod1 #0~git20210708.4beda54
Jul 11 18:26:44 pop-os kernel: [  216.337913] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4S, BIOS P3.90 01/26/2021
Jul 11 18:26:44 pop-os kernel: [  216.337914] Workqueue: btrfs-delalloc btrfs_work_helper [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.337924] RIP: 0010:kfence_handle_page_fault+0xc0/0x220
Jul 11 18:26:44 pop-os kernel: [  216.337926] Code: 89 c4 41 b8 01 00 00 00 e9 14 01 00 00 48 81 e3 00 f0 ff ff 48 89 df 31 f6 e8 bc 03 00 00 89 c1 b0 01 84 c9 0f 85 4a 01 00 00 <0f> 0b e9 4e 01 00 00 48 8d 8b 00 f0 ff ff 48 8b 05 23 b5 31 01 48
Jul 11 18:26:44 pop-os kernel: [  216.337927] RSP: 0018:ffff984981617700 EFLAGS: 00210046
Jul 11 18:26:44 pop-os kernel: [  216.337928] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.337928] RDX: 7e442c6f89015c00 RSI: 0000000000000000 RDI: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.337929] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.337929] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.337930] R13: 0000000000000000 R14: ffff9849816177c8 R15: 0000000000000002
Jul 11 18:26:44 pop-os kernel: [  216.337930] FS:  0000000000000000(0000) GS:ffff8d14eeb80000(0000) knlGS:0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.337931] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 11 18:26:44 pop-os kernel: [  216.337932] CR2: 0000000000000019 CR3: 00000001e0116000 CR4: 0000000000750ee0
Jul 11 18:26:44 pop-os kernel: [  216.337933] PKRU: 55555554
Jul 11 18:26:44 pop-os kernel: [  216.337933] Call Trace:
Jul 11 18:26:44 pop-os kernel: [  216.337935]  page_fault_oops+0x9d/0x3c0
Jul 11 18:26:44 pop-os kernel: [  216.337937]  exc_page_fault+0x6a/0xb0
Jul 11 18:26:44 pop-os kernel: [  216.337939]  asm_exc_page_fault+0x1e/0x30
Jul 11 18:26:44 pop-os kernel: [  216.337941] RIP: 0010:rt_spin_lock+0xe/0x50
Jul 11 18:26:44 pop-os kernel: [  216.337942] Code: 25 80 8b 01 00 31 d2 48 89 c8 f0 48 0f b1 57 18 48 39 c8 75 01 c3 e9 b1 fc ff ff cc 41 56 53 65 48 8b 0c 25 80 8b 01 00 31 c0 <f0> 48 0f b1 4f 18 48 85 c0 75 0d e8 12 11 4f ff 5b 41 5e e9 1a 60
Jul 11 18:26:44 pop-os kernel: [  216.337943] RSP: 0018:ffff984981617878 EFLAGS: 00210246
Jul 11 18:26:44 pop-os kernel: [  216.337944] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff8d11d6464100
Jul 11 18:26:44 pop-os kernel: [  216.337944] RDX: 0000000000000000 RSI: fffff556044fd408 RDI: 0000000000000001
Jul 11 18:26:44 pop-os kernel: [  216.337945] RBP: 0000000000000001 R08: 0000000080080008 R09: 0000000000080008
Jul 11 18:26:44 pop-os kernel: [  216.337945] R10: ffff8d11c62c72f0 R11: ffffffff9c842350 R12: 00000000ffffffff
Jul 11 18:26:44 pop-os kernel: [  216.337945] R13: 0000000000080007 R14: 0000000000080007 R15: 00000001ffffffff
Jul 11 18:26:44 pop-os kernel: [  216.337946]  ? mempool_free_slab+0x10/0x10
Jul 11 18:26:44 pop-os kernel: [  216.337948]  deactivate_slab+0x3c1/0x5c0
Jul 11 18:26:44 pop-os kernel: [  216.337950]  ___slab_alloc+0x3ac/0x540
Jul 11 18:26:44 pop-os kernel: [  216.337951]  ? mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  216.337952]  __kmalloc+0x10f/0x260
Jul 11 18:26:44 pop-os kernel: [  216.337953]  ? mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  216.337955]  mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  216.337956]  mempool_alloc+0x44/0x1c0
Jul 11 18:26:44 pop-os kernel: [  216.337958]  nvme_queue_rq+0x215/0x870 [nvme]
Jul 11 18:26:44 pop-os kernel: [  216.337960]  __blk_mq_try_issue_directly+0x14d/0x2a0
Jul 11 18:26:44 pop-os kernel: [  216.337962]  ? prepare_to_wait_exclusive+0x61/0x70
Jul 11 18:26:44 pop-os kernel: [  216.337964]  blk_mq_try_issue_directly+0x4a/0x100
Jul 11 18:26:44 pop-os kernel: [  216.337965]  blk_mq_submit_bio+0x3c6/0x4c0
Jul 11 18:26:44 pop-os kernel: [  216.337966]  submit_bio_noacct+0x3cb/0x4b0
Jul 11 18:26:44 pop-os kernel: [  216.337968]  submit_bio+0xf9/0x1c0
Jul 11 18:26:44 pop-os kernel: [  216.337970]  btrfs_map_bio+0x2b1/0x440 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.337977]  btrfs_submit_compressed_write+0x34f/0x470 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.337985]  submit_compressed_extents+0x51b/0x680 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.337992]  btrfs_work_helper+0x148/0x1e0 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.338000]  process_one_work+0x1db/0x4c0
Jul 11 18:26:44 pop-os kernel: [  216.338001]  worker_thread+0x26d/0x4a0
Jul 11 18:26:44 pop-os kernel: [  216.338002]  kthread+0x173/0x190
Jul 11 18:26:44 pop-os kernel: [  216.338004]  ? process_one_work+0x4c0/0x4c0
Jul 11 18:26:44 pop-os kernel: [  216.338004]  ? kthread_blkcg+0x30/0x30
Jul 11 18:26:44 pop-os kernel: [  216.338006]  ret_from_fork+0x22/0x30
Jul 11 18:26:44 pop-os kernel: [  216.338008] ---[ end trace 0000000000000003 ]---
Jul 11 18:26:44 pop-os kernel: [  216.379435] [drm] Fence fallback timer expired on ring gfx
Jul 11 18:26:44 pop-os kernel: [  216.655895] general protection fault, maybe for address 0xfffff5560426a018: 0000 [#1] PREEMPT_RT SMP NOPTI
Jul 11 18:26:44 pop-os kernel: [  216.655897] CPU: 11 PID: 4643 Comm: CJobMgr::m_Work Tainted: G        W         5.13.1-rt1-xanmod1 #0~git20210708.4beda54
Jul 11 18:26:44 pop-os kernel: [  216.655898] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4S, BIOS P3.90 01/26/2021
Jul 11 18:26:44 pop-os kernel: [  216.655899] RIP: 0010:___cmpxchg_double_slab+0xa3/0x170
Jul 11 18:26:44 pop-os kernel: [  216.655902] Code: 0d 48 f7 00 00 00 08 00 0f 85 82 00 00 00 b2 01 f7 c1 00 02 00 00 74 65 fb 66 0f 1f 44 00 00 eb 5c 4c 89 c9 48 89 d0 4c 89 c2 <f0> 48 0f c7 4e 20 b2 01 75 45 eb 47 48 0f ba 36 00 65 ff 0d 6d 47
Jul 11 18:26:44 pop-os kernel: [  216.655903] RSP: 0018:ffff9849826674e8 EFLAGS: 00210206
Jul 11 18:26:44 pop-os kernel: [  216.655904] RAX: ffff8d11c0042d00 RBX: ffff8d11c0042d00 RCX: ffff8d11c9a87000
Jul 11 18:26:44 pop-os kernel: [  216.655904] RDX: ffff8d11c9a87000 RSI: fffff55604269ff8 RDI: 0000000040000000
Jul 11 18:26:44 pop-os kernel: [  216.655905] RBP: ffff8d11c0042d00 R08: ffff8d11c9a87000 R09: ffff8d11c9a87000
Jul 11 18:26:44 pop-os kernel: [  216.655905] R10: ffff8d11c62c72f0 R11: ffffffff9c842350 R12: fffff5560426a000
Jul 11 18:26:44 pop-os kernel: [  216.655905] R13: ffff8d11c0042d00 R14: 00000000000049a8 R15: ffff8d11c9a87000
Jul 11 18:26:44 pop-os kernel: [  216.655906] FS:  0000000000000000(0000) GS:ffff8d14eecc0000(0063) knlGS:00000000c87feac0
Jul 11 18:26:44 pop-os kernel: [  216.655907] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
Jul 11 18:26:44 pop-os kernel: [  216.655907] CR2: 00007fde827befc8 CR3: 00000001e0116000 CR4: 0000000000750ee0
Jul 11 18:26:44 pop-os kernel: [  216.655908] PKRU: 55555554
Jul 11 18:26:44 pop-os kernel: [  216.655908] Call Trace:
Jul 11 18:26:44 pop-os kernel: [  216.655909]  get_partial_node+0xf5/0x270
Jul 11 18:26:44 pop-os kernel: [  216.655911]  ___slab_alloc+0x1f2/0x540
Jul 11 18:26:44 pop-os kernel: [  216.655912]  ? mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  216.655914]  __kmalloc+0x10f/0x260
Jul 11 18:26:44 pop-os kernel: [  216.655915]  ? mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  216.655916]  mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  216.655917]  mempool_alloc+0x44/0x1c0
Jul 11 18:26:44 pop-os kernel: [  216.655919]  nvme_queue_rq+0x215/0x870 [nvme]
Jul 11 18:26:44 pop-os kernel: [  216.655921]  ? set_next_entity+0x48/0x170
Jul 11 18:26:44 pop-os kernel: [  216.655923]  __blk_mq_try_issue_directly+0x14d/0x2a0
Jul 11 18:26:44 pop-os kernel: [  216.655924]  ? __set_cpus_allowed_ptr.llvm.15140772728589696328+0x1a1/0x820
Jul 11 18:26:44 pop-os kernel: [  216.655926]  blk_mq_try_issue_list_directly+0xce/0x200
Jul 11 18:26:44 pop-os kernel: [  216.655927]  blk_mq_sched_insert_requests+0x7f/0xf0
Jul 11 18:26:44 pop-os kernel: [  216.655928]  blk_mq_flush_plug_list+0xb8/0x150
Jul 11 18:26:44 pop-os kernel: [  216.655929]  blk_flush_plug_list+0xc3/0xf0
Jul 11 18:26:44 pop-os kernel: [  216.655931]  blk_mq_submit_bio+0x2ce/0x4c0
Jul 11 18:26:44 pop-os kernel: [  216.655932]  submit_bio_noacct+0x3cb/0x4b0
Jul 11 18:26:44 pop-os kernel: [  216.655934]  submit_bio+0xf9/0x1c0
Jul 11 18:26:44 pop-os kernel: [  216.655935]  btrfs_map_bio+0x2b1/0x440 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.655943]  btrfs_submit_data_bio+0x169/0x1d0 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.655949]  submit_extent_page+0x106/0x370 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.655956]  __extent_writepage_io+0x235/0x390 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.655962]  ? __unlock_for_delalloc+0x30/0x30 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.655968]  __extent_writepage+0x265/0x320 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.655975]  extent_writepages+0x368/0x5a0 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.655981]  ? rt_spin_lock+0x32/0x50
Jul 11 18:26:44 pop-os kernel: [  216.655982]  ? ___slab_alloc+0x27f/0x540
Jul 11 18:26:44 pop-os kernel: [  216.655984]  do_writepages+0x4b/0xf0
Jul 11 18:26:44 pop-os kernel: [  216.655985]  __filemap_fdatawrite_range+0xf9/0x130
Jul 11 18:26:44 pop-os kernel: [  216.655987]  btrfs_sync_file+0xb7/0x4d0 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.655994]  __ia32_sys_fdatasync+0x46/0x80
Jul 11 18:26:44 pop-os kernel: [  216.655995]  do_int80_syscall_32+0x53/0x90
Jul 11 18:26:44 pop-os kernel: [  216.655996]  entry_INT80_compat+0x85/0x8a
Jul 11 18:26:44 pop-os kernel: [  216.655997] RIP: 0023:0xf7f95092
Jul 11 18:26:44 pop-os kernel: [  216.655998] Code: 00 00 66 0f 1f 44 00 00 f3 0f 1e fb ff a3 14 00 00 00 66 0f 1f 44 00 00 f3 0f 1e fb ff a3 18 00 00 00 66 0f 1f 44 00 00 cd 80 <c3> 8d b4 26 00 00 00 00 8d b6 00 00 00 00 8b 1c 24 c3 8d b4 26 00
Jul 11 18:26:44 pop-os kernel: [  216.655999] RSP: 002b:00000000c87fc92c EFLAGS: 00200286 ORIG_RAX: 0000000000000094
Jul 11 18:26:44 pop-os kernel: [  216.655999] RAX: ffffffffffffffda RBX: 000000000000008c RCX: 0000000000000002
Jul 11 18:26:44 pop-os kernel: [  216.656000] RDX: 0000000000000000 RSI: 00000000f7b24000 RDI: 00000000c87fcd68
Jul 11 18:26:44 pop-os kernel: [  216.656000] RBP: 0000000000080c6e R08: 0000000000000000 R09: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.656001] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.656001] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.656002] Modules linked in: snd_seq_dummy snd_hrtimer uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_usb_audio videodev intel_rapl_msr snd_usbmidi_lib mc intel_rapl_common nls_iso8859_1 edac_mce_amd input_leds joydev snd_hda_codec_realtek xpad kvm snd_hda_codec_generic ff_memless ledtrig_audio snd_hda_codec_hdmi rapl snd_hda_intel snd_intel_dspcfg snd_seq_midi snd_seq_midi_event snd_intel_sdw_acpi wmi_bmof snd_rawmidi snd_hda_codec snd_hda_core snd_seq snd_hwdep snd_seq_device snd_pcm ccp snd_timer k10temp snd soundcore mac_hid sch_fq_codel msr parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs dm_crypt raid10 raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx raid6_pq xor libcrc32c raid1 raid0 multipath linear system76_acpi hid_generic usbhid hid amdgpu iommu_v2 gpu_sched drm_ttm_helper ttm drm_kms_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cec aesni_intel rc_core sysimgblt igb syscopyarea crypto_simd ahci
Jul 11 18:26:44 pop-os kernel: [  216.656025]  sysfillrect fb_sys_fops xhci_pci dca cryptd libahci i2c_piix4 xhci_pci_renesas i2c_algo_bit drm nvme nvme_core wmi
Jul 11 18:26:44 pop-os kernel: [  216.656055] ---[ end trace 0000000000000004 ]---
Jul 11 18:26:44 pop-os kernel: [  216.656063] ------------[ cut here ]------------
Jul 11 18:26:44 pop-os kernel: [  216.656064] WARNING: CPU: 11 PID: 4643 at rcu_note_context_switch+0x188/0x390
Jul 11 18:26:44 pop-os kernel: [  216.656066] Modules linked in: snd_seq_dummy snd_hrtimer uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_usb_audio videodev intel_rapl_msr snd_usbmidi_lib mc intel_rapl_common nls_iso8859_1 edac_mce_amd input_leds joydev snd_hda_codec_realtek xpad kvm snd_hda_codec_generic ff_memless ledtrig_audio snd_hda_codec_hdmi rapl snd_hda_intel snd_intel_dspcfg snd_seq_midi snd_seq_midi_event snd_intel_sdw_acpi wmi_bmof snd_rawmidi snd_hda_codec snd_hda_core snd_seq snd_hwdep snd_seq_device snd_pcm ccp snd_timer k10temp snd soundcore mac_hid sch_fq_codel msr parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs dm_crypt raid10 raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx raid6_pq xor libcrc32c raid1 raid0 multipath linear system76_acpi hid_generic usbhid hid amdgpu iommu_v2 gpu_sched drm_ttm_helper ttm drm_kms_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cec aesni_intel rc_core sysimgblt igb syscopyarea crypto_simd ahci
Jul 11 18:26:44 pop-os kernel: [  216.656077]  sysfillrect fb_sys_fops xhci_pci dca cryptd libahci i2c_piix4 xhci_pci_renesas i2c_algo_bit drm nvme nvme_core wmi
Jul 11 18:26:44 pop-os kernel: [  216.656079] CPU: 11 PID: 4643 Comm: CJobMgr::m_Work Tainted: G      D W         5.13.1-rt1-xanmod1 #0~git20210708.4beda54
Jul 11 18:26:44 pop-os kernel: [  216.656080] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4S, BIOS P3.90 01/26/2021
Jul 11 18:26:44 pop-os kernel: [  216.656080] RIP: 0010:rcu_note_context_switch+0x188/0x390
Jul 11 18:26:44 pop-os kernel: [  216.656081] Code: 20 75 44 83 fa 07 7f 46 83 fa 03 7f 0a 83 c2 ff 83 fa 03 72 72 eb 52 83 fa 05 7f 07 83 fa 04 74 48 eb 56 83 fa 06 74 41 eb 4f <0f> 0b e9 a9 fe ff ff 0f 0b e9 2a ff ff ff 0f 0b e9 3a ff ff ff 0f
Jul 11 18:26:44 pop-os kernel: [  216.656082] RSP: 0018:ffff9849826671c0 EFLAGS: 00210002
Jul 11 18:26:44 pop-os kernel: [  216.656082] RAX: 0000000000000001 RBX: ffff8d14eece1f00 RCX: ffff984982667eb8
Jul 11 18:26:44 pop-os kernel: [  216.656083] RDX: 000000000000000b RSI: 0000000000200286 RDI: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.656083] RBP: 00000000000003e8 R08: 0000000000000000 R09: ffffffff9de8ee70
Jul 11 18:26:44 pop-os kernel: [  216.656083] R10: 0000000000000000 R11: ffff984982667288 R12: ffff8d11c4c42080
Jul 11 18:26:44 pop-os kernel: [  216.656084] R13: 00000000000000b9 R14: 0000000000000000 R15: ffff8d11c4c42080
Jul 11 18:26:44 pop-os kernel: [  216.656084] FS:  0000000000000000(0000) GS:ffff8d14eecc0000(0063) knlGS:00000000c87feac0
Jul 11 18:26:44 pop-os kernel: [  216.656085] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
Jul 11 18:26:44 pop-os kernel: [  216.656085] CR2: 00007fde827befc8 CR3: 00000001e0116000 CR4: 0000000000750ee0
Jul 11 18:26:44 pop-os kernel: [  216.656086] PKRU: 55555554
Jul 11 18:26:44 pop-os kernel: [  216.656086] Call Trace:
Jul 11 18:26:44 pop-os kernel: [  216.656087]  __schedule+0x64/0x610
Jul 11 18:26:44 pop-os kernel: [  216.656088]  schedule+0x7e/0xc0
Jul 11 18:26:44 pop-os kernel: [  216.656089]  schedule_timeout+0xa5/0x110
Jul 11 18:26:44 pop-os kernel: [  216.656090]  ? update_process_times+0xb0/0xb0
Jul 11 18:26:44 pop-os kernel: [  216.656091]  msleep+0x80/0x90
Jul 11 18:26:44 pop-os kernel: [  216.656092]  pr_flush+0x1df/0x3a0
Jul 11 18:26:44 pop-os kernel: [  216.656094]  oops_exit+0x34/0x40
Jul 11 18:26:44 pop-os kernel: [  216.656095]  oops_end+0x69/0x100
Jul 11 18:26:44 pop-os kernel: [  216.656097]  exc_general_protection+0x26e/0x3d0
Jul 11 18:26:44 pop-os kernel: [  216.656099]  ? prep_new_page+0x8e/0x200
Jul 11 18:26:44 pop-os kernel: [  216.656100]  asm_exc_general_protection+0x1e/0x30
Jul 11 18:26:44 pop-os kernel: [  216.656101] RIP: 0010:___cmpxchg_double_slab+0xa3/0x170
Jul 11 18:26:44 pop-os kernel: [  216.656102] Code: 0d 48 f7 00 00 00 08 00 0f 85 82 00 00 00 b2 01 f7 c1 00 02 00 00 74 65 fb 66 0f 1f 44 00 00 eb 5c 4c 89 c9 48 89 d0 4c 89 c2 <f0> 48 0f c7 4e 20 b2 01 75 45 eb 47 48 0f ba 36 00 65 ff 0d 6d 47
Jul 11 18:26:44 pop-os kernel: [  216.656102] RSP: 0018:ffff9849826674e8 EFLAGS: 00210206
Jul 11 18:26:44 pop-os kernel: [  216.656103] RAX: ffff8d11c0042d00 RBX: ffff8d11c0042d00 RCX: ffff8d11c9a87000
Jul 11 18:26:44 pop-os kernel: [  216.656103] RDX: ffff8d11c9a87000 RSI: fffff55604269ff8 RDI: 0000000040000000
Jul 11 18:26:44 pop-os kernel: [  216.656103] RBP: ffff8d11c0042d00 R08: ffff8d11c9a87000 R09: ffff8d11c9a87000
Jul 11 18:26:44 pop-os kernel: [  216.656104] R10: ffff8d11c62c72f0 R11: ffffffff9c842350 R12: fffff5560426a000
Jul 11 18:26:44 pop-os kernel: [  216.656104] R13: ffff8d11c0042d00 R14: 00000000000049a8 R15: ffff8d11c9a87000
Jul 11 18:26:44 pop-os kernel: [  216.656104]  ? mempool_free_slab+0x10/0x10
Jul 11 18:26:44 pop-os kernel: [  216.656106]  get_partial_node+0xf5/0x270
Jul 11 18:26:44 pop-os kernel: [  216.656107]  ___slab_alloc+0x1f2/0x540
Jul 11 18:26:44 pop-os kernel: [  216.656108]  ? mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  216.656109]  __kmalloc+0x10f/0x260
Jul 11 18:26:44 pop-os kernel: [  216.656110]  ? mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  216.656111]  mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  216.656112]  mempool_alloc+0x44/0x1c0
Jul 11 18:26:44 pop-os kernel: [  216.656113]  nvme_queue_rq+0x215/0x870 [nvme]
Jul 11 18:26:44 pop-os kernel: [  216.656114]  ? set_next_entity+0x48/0x170
Jul 11 18:26:44 pop-os kernel: [  216.656116]  __blk_mq_try_issue_directly+0x14d/0x2a0
Jul 11 18:26:44 pop-os kernel: [  216.656117]  ? __set_cpus_allowed_ptr.llvm.15140772728589696328+0x1a1/0x820
Jul 11 18:26:44 pop-os kernel: [  216.656118]  blk_mq_try_issue_list_directly+0xce/0x200
Jul 11 18:26:44 pop-os kernel: [  216.656119]  blk_mq_sched_insert_requests+0x7f/0xf0
Jul 11 18:26:44 pop-os kernel: [  216.656120]  blk_mq_flush_plug_list+0xb8/0x150
Jul 11 18:26:44 pop-os kernel: [  216.656121]  blk_flush_plug_list+0xc3/0xf0
Jul 11 18:26:44 pop-os kernel: [  216.656123]  blk_mq_submit_bio+0x2ce/0x4c0
Jul 11 18:26:44 pop-os kernel: [  216.656124]  submit_bio_noacct+0x3cb/0x4b0
Jul 11 18:26:44 pop-os kernel: [  216.656125]  submit_bio+0xf9/0x1c0
Jul 11 18:26:44 pop-os kernel: [  216.656126]  btrfs_map_bio+0x2b1/0x440 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.656132]  btrfs_submit_data_bio+0x169/0x1d0 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.656139]  submit_extent_page+0x106/0x370 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.656145]  __extent_writepage_io+0x235/0x390 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.656151]  ? __unlock_for_delalloc+0x30/0x30 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.656157]  __extent_writepage+0x265/0x320 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.656163]  extent_writepages+0x368/0x5a0 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.656170]  ? rt_spin_lock+0x32/0x50
Jul 11 18:26:44 pop-os kernel: [  216.656171]  ? ___slab_alloc+0x27f/0x540
Jul 11 18:26:44 pop-os kernel: [  216.656172]  do_writepages+0x4b/0xf0
Jul 11 18:26:44 pop-os kernel: [  216.656173]  __filemap_fdatawrite_range+0xf9/0x130
Jul 11 18:26:44 pop-os kernel: [  216.656174]  btrfs_sync_file+0xb7/0x4d0 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  216.656181]  __ia32_sys_fdatasync+0x46/0x80
Jul 11 18:26:44 pop-os kernel: [  216.656182]  do_int80_syscall_32+0x53/0x90
Jul 11 18:26:44 pop-os kernel: [  216.656182]  entry_INT80_compat+0x85/0x8a
Jul 11 18:26:44 pop-os kernel: [  216.656183] RIP: 0023:0xf7f95092
Jul 11 18:26:44 pop-os kernel: [  216.656184] Code: 00 00 66 0f 1f 44 00 00 f3 0f 1e fb ff a3 14 00 00 00 66 0f 1f 44 00 00 f3 0f 1e fb ff a3 18 00 00 00 66 0f 1f 44 00 00 cd 80 <c3> 8d b4 26 00 00 00 00 8d b6 00 00 00 00 8b 1c 24 c3 8d b4 26 00
Jul 11 18:26:44 pop-os kernel: [  216.656184] RSP: 002b:00000000c87fc92c EFLAGS: 00200286 ORIG_RAX: 0000000000000094
Jul 11 18:26:44 pop-os kernel: [  216.656185] RAX: ffffffffffffffda RBX: 000000000000008c RCX: 0000000000000002
Jul 11 18:26:44 pop-os kernel: [  216.656185] RDX: 0000000000000000 RSI: 00000000f7b24000 RDI: 00000000c87fcd68
Jul 11 18:26:44 pop-os kernel: [  216.656185] RBP: 0000000000080c6e R08: 0000000000000000 R09: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.656186] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.656186] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
Jul 11 18:26:44 pop-os kernel: [  216.656187] ---[ end trace 0000000000000005 ]---
Jul 11 18:26:44 pop-os kernel: [  217.338162] BUG: kernel NULL pointer dereference, address: 0000000000000019
Jul 11 18:26:44 pop-os kernel: [  217.338162] #PF: supervisor write access in kernel mode
Jul 11 18:26:44 pop-os kernel: [  217.338163] #PF: error_code(0x0002) - not-present page
Jul 11 18:26:44 pop-os kernel: [  217.338164] PGD 1ab2f1067 P4D 1ab2f1067 PUD 0 
Jul 11 18:26:44 pop-os kernel: [  217.338165] Oops: 0002 [#2] PREEMPT_RT SMP NOPTI
Jul 11 18:26:44 pop-os kernel: [  217.338166] CPU: 6 PID: 374 Comm: kworker/u64:7 Tainted: G      D W         5.13.1-rt1-xanmod1 #0~git20210708.4beda54
Jul 11 18:26:44 pop-os kernel: [  217.338168] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X570 Phantom Gaming 4S, BIOS P3.90 01/26/2021
Jul 11 18:26:44 pop-os kernel: [  217.338168] Workqueue: btrfs-delalloc btrfs_work_helper [btrfs]
Jul 11 18:26:44 pop-os kernel: [  217.338175] RIP: 0010:rt_spin_lock+0xe/0x50
Jul 11 18:26:44 pop-os kernel: [  217.338177] Code: 25 80 8b 01 00 31 d2 48 89 c8 f0 48 0f b1 57 18 48 39 c8 75 01 c3 e9 b1 fc ff ff cc 41 56 53 65 48 8b 0c 25 80 8b 01 00 31 c0 <f0> 48 0f b1 4f 18 48 85 c0 75 0d e8 12 11 4f ff 5b 41 5e e9 1a 60
Jul 11 18:26:44 pop-os kernel: [  217.338177] RSP: 0018:ffff984981617878 EFLAGS: 00210246
Jul 11 18:26:44 pop-os kernel: [  217.338178] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff8d11d6464100
Jul 11 18:26:44 pop-os kernel: [  217.338179] RDX: 0000000000000000 RSI: fffff556044fd408 RDI: 0000000000000001
Jul 11 18:26:44 pop-os kernel: [  217.338179] RBP: 0000000000000001 R08: 0000000080080008 R09: 0000000000080008
Jul 11 18:26:44 pop-os kernel: [  217.338180] R10: ffff8d11c62c72f0 R11: ffffffff9c842350 R12: 00000000ffffffff
Jul 11 18:26:44 pop-os kernel: [  217.338180] R13: 0000000000080007 R14: 0000000000080007 R15: 00000001ffffffff
Jul 11 18:26:44 pop-os kernel: [  217.338181] FS:  0000000000000000(0000) GS:ffff8d14eeb80000(0000) knlGS:0000000000000000
Jul 11 18:26:44 pop-os kernel: [  217.338181] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 11 18:26:44 pop-os kernel: [  217.338182] CR2: 0000000000000019 CR3: 00000001e0116000 CR4: 0000000000750ee0
Jul 11 18:26:44 pop-os kernel: [  217.338183] PKRU: 55555554
Jul 11 18:26:44 pop-os kernel: [  217.338183] Call Trace:
Jul 11 18:26:44 pop-os kernel: [  217.338183]  deactivate_slab+0x3c1/0x5c0
Jul 11 18:26:44 pop-os kernel: [  217.338185]  ___slab_alloc+0x3ac/0x540
Jul 11 18:26:44 pop-os kernel: [  217.338186]  ? mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  217.338188]  __kmalloc+0x10f/0x260
Jul 11 18:26:44 pop-os kernel: [  217.338188]  ? mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  217.338190]  mempool_kmalloc+0xc/0x10
Jul 11 18:26:44 pop-os kernel: [  217.338191]  mempool_alloc+0x44/0x1c0
Jul 11 18:26:44 pop-os kernel: [  217.338192]  nvme_queue_rq+0x215/0x870 [nvme]
Jul 11 18:26:44 pop-os kernel: [  217.338195]  __blk_mq_try_issue_directly+0x14d/0x2a0
Jul 11 18:26:44 pop-os kernel: [  217.338196]  ? prepare_to_wait_exclusive+0x61/0x70
Jul 11 18:26:44 pop-os kernel: [  217.338197]  blk_mq_try_issue_directly+0x4a/0x100
Jul 11 18:26:44 pop-os kernel: [  217.338198]  blk_mq_submit_bio+0x3c6/0x4c0
Jul 11 18:26:44 pop-os kernel: [  217.338200]  submit_bio_noacct+0x3cb/0x4b0
Jul 11 18:26:44 pop-os kernel: [  217.338201]  submit_bio+0xf9/0x1c0
Jul 11 18:26:44 pop-os kernel: [  217.338203]  btrfs_map_bio+0x2b1/0x440 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  217.338210]  btrfs_submit_compressed_write+0x34f/0x470 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  217.338217]  submit_compressed_extents+0x51b/0x680 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  217.338224]  btrfs_work_helper+0x148/0x1e0 [btrfs]
Jul 11 18:26:44 pop-os kernel: [  217.338231]  process_one_work+0x1db/0x4c0
Jul 11 18:26:44 pop-os kernel: [  217.338233]  worker_thread+0x26d/0x4a0
Jul 11 18:26:44 pop-os kernel: [  217.338234]  kthread+0x173/0x190
Jul 11 18:26:44 pop-os kernel: [  217.338235]  ? process_one_work+0x4c0/0x4c0
Jul 11 18:26:44 pop-os kernel: [  217.338236]  ? kthread_blkcg+0x30/0x30
Jul 11 18:26:44 pop-os kernel: [  217.338237]  ret_from_fork+0x22/0x30
Jul 11 18:26:44 pop-os kernel: [  217.338239] Modules linked in: snd_seq_dummy snd_hrtimer uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common snd_usb_audio videodev intel_rapl_msr snd_usbmidi_lib mc intel_rapl_common nls_iso8859_1 edac_mce_amd input_leds joydev snd_hda_codec_realtek xpad kvm snd_hda_codec_generic ff_memless ledtrig_audio snd_hda_codec_hdmi rapl snd_hda_intel snd_intel_dspcfg snd_seq_midi snd_seq_midi_event snd_intel_sdw_acpi wmi_bmof snd_rawmidi snd_hda_codec snd_hda_core snd_seq snd_hwdep snd_seq_device snd_pcm ccp snd_timer k10temp snd soundcore mac_hid sch_fq_codel msr parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs dm_crypt raid10 raid456 async_raid6_recov async_pq async_xor async_memcpy async_tx raid6_pq xor libcrc32c raid1 raid0 multipath linear system76_acpi hid_generic usbhid hid amdgpu iommu_v2 gpu_sched drm_ttm_helper ttm drm_kms_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel cec aesni_intel rc_core sysimgblt igb syscopyarea crypto_simd ahci
Jul 11 18:26:44 pop-os kernel: [  217.338256]  sysfillrect fb_sys_fops xhci_pci dca cryptd libahci i2c_piix4 xhci_pci_renesas i2c_algo_bit drm nvme nvme_core wmi
Jul 11 18:26:44 pop-os kernel: [  217.338258] CR2: 0000000000000019
Jul 11 18:26:44 pop-os kernel: [  217.338259] ---[ end trace 0000000000000006 ]---

Please let me know if anything else is needed.

Thanks, cj0nes

EDIT: I just realized that the kernel I updated to is classified as "In Development". I did not notice that before I updated. Well, I guess this issue could possibly aid development then.

cj0nes avatar Jul 11 '21 23:07 cj0nes

Hi,

I am also experiencing system freezes and crashes on pop-os with the production ready 5.13 kernel. The system completely freezes (unable to do anything, SSH won't work either). Sometimes it will also completely crash after being frozen for a couple of seconds and reboot the system.

How has the above log been obtained? So that I could also supply one in hopes this get fixed. At the moment the freeze/crash happens about once per day, this is obviously something that needs to be fixed.

Thanks

Shadow505 avatar Sep 01 '21 14:09 Shadow505

How has the above log been obtained?

I believe I pulled it from /var/log/kern.log

cj0nes avatar Sep 03 '21 02:09 cj0nes

@cj0nes: Thanks! I will check it the next time the system dead-freezes and/or crashed. Hopefully this can be resolved.

Shadow505 avatar Sep 03 '21 08:09 Shadow505

The dead-freeze just happend again. I looked through syslog, dmesg, kern.log and the journal. None of those contain any logs close to the date/time of the freeze.. Not sure what to do here. Any idea @xanmod?

@cj0nes: Are you using BTRFS by any chance?

Shadow505 avatar Sep 03 '21 13:09 Shadow505

@xanmod:

I am quite certain that the issue has something to do with either BTRFS specifically or the filesystem in general. The freeze happend exactly as I was saving a text file. I just opened up that file again and it suddenly is completely empty. It was not empty originally, then I added a line of text, saved it, then the PC froze, I hard-rebooted it (via the reset button on the case) and now the file is completely empty.

If this cannot be resolved asap, I have to leave this kernel because data loss is obviously the worst that can happen and unfortunately did happen.

Shadow505 avatar Sep 03 '21 13:09 Shadow505

Well, apparently much more than a single file has been corrupted. It broke multiple applications (they won't start anymore), several Wine applications (including Office 365). Definitely have to move away from this kernel for now.

Shadow505 avatar Sep 03 '21 13:09 Shadow505

@cj0nes: Are you using BTRFS by any chance?

Yes, I am actually. Is there any particular output I can share that would help?

cj0nes avatar Sep 03 '21 20:09 cj0nes

Yeah, that pretty much confirms my suspicion that the issue has something to do with BTRFS. I do not experience this issues when using the stock kernel, so there is most certainly something wrong with the BTRFS implementation in the Xanmod kernel.

Hopefully @xanmod will be able to sort this out. For now I have switched back to the stock kernel to avoid any further data loss.

Shadow505 avatar Sep 05 '21 08:09 Shadow505

@Shadow505, I fell back to 5.11.12-rt11-xanmod1, if that's any help for narrowing down what the issue could be in the newer Xanmod kernel.

cj0nes avatar Sep 06 '21 23:09 cj0nes