openpilot
openpilot copied to clipboard
Published
20 hours ago •
commaai
commaai
C3 restarts randomly due to camera error
Describe the bug
My C3 has been restarting randomly even while onroad. I'm opening this issue since the C3 works perfectly fine, no "camera mounted" error. I think maybe this can be a software fix
Context:
- AGNOS 5.1,
master-ci - NVME remount was suggested and done, but the issue persists
I managed to change /var/log to /data/log to persist the log. I got a couple of crashes logged in the file attached
The crash happens on Jun 25 10:33:50
- syslog
Jun 25 10:33:57 tici kernel: [10123.330454] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 41 old ctx ffffff825b6201a8 cookie 136706 new ctx ffffff825b629228 cookie 136719
kernlogbefore the crash
Jun 25 10:40:00 tici kernel: [ 56.432872] CAM_INFO: CAM-CRM: cam_req_mgr_open: 130 cam_req_mgr_open:130 disalbe lpm
Jun 25 10:40:00 tici kernel: [ 56.432894] CAM_INFO: CAM-SENSOR: cam_pm_qos_update_request: 117 cam_pm_qos_update_request: update request 100
Jun 25 10:40:00 tici kernel: [ 56.489177] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 768 Probe success,slot:2,slave_addr:0x20,sensor_id:0x354
Jun 25 10:40:00 tici kernel: [ 56.548856] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 841 CAM_ACQUIRE_DEV Success, sensor_id:0x354,sensor_slave_addr:0x20
Jun 25 10:40:00 tici kernel: [ 56.618333] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 768 Probe success,slot:1,slave_addr:0x30,sensor_id:0x354
Jun 25 10:40:00 tici kernel: [ 56.624773] CAM_INFO: CAM: cam_res_mgr_shared_pinctrl_put: 273 Need hold put this pinctrl
Jun 25 10:40:00 tici kernel: [ 56.678142] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 841 CAM_ACQUIRE_DEV Success, sensor_id:0x354,sensor_slave_addr:0x30
Jun 25 10:40:00 tici kernel: [ 56.748364] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 768 Probe success,slot:0,slave_addr:0x20,sensor_id:0x354
Jun 25 10:40:00 tici kernel: [ 56.754807] CAM_INFO: CAM: cam_res_mgr_shared_pinctrl_put: 273 Need hold put this pinctrl
Jun 25 10:40:00 tici kernel: [ 56.808180] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 841 CAM_ACQUIRE_DEV Success, sensor_id:0x354,sensor_slave_addr:0x20
Jun 25 10:40:00 tici kernel: [ 56.836510] msm_vidc: info: Opening video instance: 0000000000000000, 0
Jun 25 10:40:00 tici kernel: [ 56.842398] msm_vidc: info: Opening video instance: 0000000000000000, 0
Jun 25 10:40:00 tici kernel: [ 56.845906] msm_vidc: info: Opening video instance: 0000000000000000, 0
Jun 25 10:40:00 tici kernel: [ 56.874454] msm_vidc: info: Opening video instance: 0000000000000000, 0
Jun 25 10:40:01 tici kernel: [ 57.122544] WARNING: CPU: 3 PID: 52540 at vb2_warn_zero_bytesused.isra.10.part.11+0x24/0x68
Jun 25 10:40:01 tici kernel: [ 57.122581] CPU: 3 PID: 52540 Comm: dcamera.hevc Tainted: G C E 4.9.103+ #3
Jun 25 10:40:01 tici kernel: [ 57.122598] task: fffffff025bd8e00 task.stack: ffffffefd71e4000
Jun 25 10:40:01 tici kernel: [ 57.122618] LR is at __fill_vb2_buffer+0x4d0/0x508
Jun 25 10:40:01 tici kernel: [ 57.122634] sp : ffffffefd71e77e0
Jun 25 10:40:01 tici kernel: [ 57.122665] x27: 0000000000000030 x26: ffffffef8d5cb400
Jun 25 10:40:01 tici kernel: [ 57.122709] x23: 0000000000000000 x22: ffffffefd71e78e8
Jun 25 10:40:01 tici kernel: [ 57.122752] x19: fffffff02d319000 x18: 0000000000000001
Jun 25 10:40:01 tici kernel: [ 57.122796] x15: 00009c7581e9aa53 x14: 0000000000000000
Jun 25 10:40:01 tici kernel: [ 57.122839] x11: 0000000000000000 x10: 0000000000000000
Jun 25 10:40:01 tici kernel: [ 57.122883] x7 : 0000000000000000 x6 : 000000000000003f
Jun 25 10:40:01 tici kernel: [ 57.122926] x3 : 0000000000000000 x2 : 0000000000000008
Jun 25 10:40:01 tici kernel: [ 57.122970]
Jun 25 10:40:01 tici kernel: [ 57.122970] PC: 0xffffff950329907c:
Jun 25 10:40:01 tici kernel: [ 57.123050] 909c 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f d0012980 52800021 392f7001
Jun 25 10:40:01 tici kernel: [ 57.123190] 90dc 91014000 97dcee62 f9400bf3 a8c27bfd d65f03c0 b0006e80 91022000 97dcee5c
Jun 25 10:40:01 tici kernel: [ 57.123268] 9590 b9403e60 17ffff3a b9400a60 35ffe480 b9401b20 b9001720 17ffff22 12802701
Jun 25 10:40:01 tici kernel: [ 57.123409] 95d0 f9400280 39406000 3617f100 b8776b00 17ffff49 aa1403e0 97fffeac f9400280
Jun 25 10:40:01 tici kernel: [ 57.123550]
Jun 25 10:40:01 tici kernel: [ 57.123550] SP: 0xffffffefd71e77a0:
Jun 25 10:40:01 tici kernel: [ 57.123627] 77c0 2c877c00 fffffff0 00000096 00000000 ffffffff 0000007f 18801120 ffffff80
Jun 25 10:40:01 tici kernel: [ 57.123767] 7800 d71e7860 ffffffef 03292df4 ffffff95 00000001 00000000 2d319000 fffffff0
Jun 25 10:40:01 tici kernel: [ 57.123847] Call trace:
Jun 25 10:40:01 tici kernel: [ 57.123868] 75e0: fffffff02d319000 0000007fffffffff ffffffefd71e77e0 ffffff95032990bc
Jun 25 10:40:01 tici kernel: [ 57.123888] 7620: ffffffefd71e7650 ffffff950288ebbc ffffff9504c88788 ffffff9503a4a024
Jun 25 10:40:01 tici kernel: [ 57.123908] 7660: 0000000000000001 ffffff950294cee4 ffffffefd71e76b0 ffffff9500000000
Jun 25 10:40:01 tici kernel: [ 57.123927] 76a0: ffffff95ffffffff 09bbccacb7439700 ffffff95057cb000 0000000000000001
Jun 25 10:40:01 tici kernel: [ 57.123946] 76e0: 000000000000003f 0000000000000000 ffffffefd71e7918 0000000000000000
Jun 25 10:40:01 tici kernel: [ 57.123955] 7700: 0000000000000000 0000000000000000
Jun 25 10:40:01 tici kernel: [ 57.123965] [<ffffff95032990bc>] vb2_warn_zero_bytesused.isra.10.part.11+0x24/0x68
Jun 25 10:40:01 tici kernel: [ 57.123975] [<ffffff95032995d0>] __fill_vb2_buffer+0x4d0/0x508
Jun 25 10:40:01 tici kernel: [ 57.123985] [<ffffff9503292df4>] __qbuf_userptr+0x384/0x6d0
Jun 25 10:40:01 tici kernel: [ 57.123994] [<ffffff9503294fb8>] __buf_prepare+0x118/0x208
Jun 25 10:40:01 tici kernel: [ 57.124004] [<ffffff95032951f8>] vb2_core_qbuf+0x78/0x2a8
Jun 25 10:40:01 tici kernel: [ 57.124013] [<ffffff9503298d00>] vb2_qbuf+0x68/0xb0
Jun 25 10:40:01 tici kernel: [ 57.124025] [<ffffff95032cb534>] msm_vidc_qbuf+0x17c/0x2f8
Jun 25 10:40:01 tici kernel: [ 57.124036] [<ffffff95032b1ae8>] msm_v4l2_qbuf+0x38/0x48
Jun 25 10:40:01 tici kernel: [ 57.124047] [<ffffff950327c8d4>] v4l_qbuf+0x4c/0x60
Jun 25 10:40:01 tici kernel: [ 57.124056] [<ffffff950327ba00>] __video_do_ioctl+0x210/0x2d0
Jun 25 10:40:01 tici kernel: [ 57.124065] [<ffffff950327b5bc>] video_usercopy+0x43c/0x620
Jun 25 10:40:01 tici kernel: [ 57.124074] [<ffffff950327b7dc>] video_ioctl2+0x3c/0x50
Jun 25 10:40:01 tici kernel: [ 57.124086] [<ffffff950327626c>] v4l2_ioctl+0xbc/0x108
Jun 25 10:40:01 tici kernel: [ 57.124100] [<ffffff9502a54b84>] do_vfs_ioctl+0xc4/0x810
Jun 25 10:40:01 tici kernel: [ 57.124109] [<ffffff9502a55360>] SyS_ioctl+0x90/0xa8
Jun 25 10:40:01 tici kernel: [ 57.124121] [<ffffff9502883f80>] el0_svc_naked+0x34/0x38
Jun 25 10:40:01 tici kernel: [ 57.124131] use of bytesused == 0 is deprecated and will be removed in the future,
Jun 25 10:40:01 tici kernel: [ 57.124143] use the actual size instead.
Jun 25 10:40:02 tici kernel: [ 58.494989] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 34 old ctx ffffff950581b968 cookie 19 new ctx ffffff95058128e8 cookie 32
Jun 25 10:40:05 tici kernel: [ 60.944873] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 9 old ctx ffffff9505817128 cookie 68 new ctx ffffff9505817128 cookie 81
Jun 25 10:40:28 tici kernel: [ 83.045224] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 53 old ctx ffffff9505817128 cookie 510 new ctx ffffff9505817128 cookie 523
Jun 25 10:40:28 tici kernel: [ 83.695290] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 58 old ctx ffffff950581b968 cookie 523 new ctx ffffff950581b968 cookie 536
Jun 25 10:40:29 tici kernel: [ 84.545298] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 6 old ctx ffffff95058128e8 cookie 541 new ctx ffffff950581b968 cookie 553
Jun 25 10:40:33 tici kernel: [ 88.445322] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 26 old ctx ffffff9505817128 cookie 618 new ctx ffffff95058128e8 cookie 631
Jun 25 10:40:57 tici kernel: [ 112.945776] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 39 old ctx ffffff95058128e8 cookie 1109 new ctx ffffff950581b968 cookie 1121
Jun 25 10:41:03 tici kernel: [ 118.545773] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 63 old ctx ffffff9505817128 cookie 1220 new ctx ffffff950581b968 cookie 1233
Jun 25 10:41:03 tici kernel: [ 118.845855] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 31 old ctx ffffff9505817128 cookie 1226 new ctx ffffff950581b968 cookie 1239
Jun 25 10:41:11 tici kernel: [ 126.345965] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 1 old ctx ffffff950581b968 cookie 1376 new ctx ffffff9505817128 cookie 1389
Jun 25 10:41:18 tici kernel: [ 133.195993] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 24 old ctx ffffff9505817128 cookie 1513 new ctx ffffff95058128e8 cookie 1526
Jun 25 10:41:26 tici kernel: [ 141.896140] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 38 old ctx ffffff950581b968 cookie 1687 new ctx ffffff950581b968 cookie 1700
Jun 25 10:41:31 tici kernel: [ 146.239265] R0: [cds_mc_thread][2964322729] [03:41:31.199801] wlan: [51976:W :SME] No AP with 11d Country code is present in scan list
Jun 25 10:41:33 tici kernel: [ 148.596265] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 21 old ctx ffffff950581b968 cookie 1821 new ctx ffffff95058128e8 cookie 1834
Jun 25 10:41:35 tici kernel: [ 150.846278] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 63 old ctx ffffff950581b968 cookie 1866 new ctx ffffff950581b968 cookie 1879
Jun 25 10:41:38 tici kernel: [ 153.246561] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 33 old ctx ffffff95058128e8 cookie 1915 new ctx ffffff950581b968 cookie 1927
Jun 25 10:41:55 tici kernel: [ 170.046763] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 6 old ctx ffffff950581b968 cookie 2251 new ctx ffffff9505817128 cookie 2263
Jun 25 10:42:06 tici kernel: [ 181.596928] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 51 old ctx ffffff9505817128 cookie 2482 new ctx ffffff950581b968 cookie 2494
Jun 25 10:42:06 tici kernel: [ 181.646786] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 56 old ctx ffffff950581b968 cookie 2482 new ctx ffffff95058128e8 cookie 2495
Jun 25 10:42:07 tici kernel: [ 182.196946] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 31 old ctx ffffff9505817128 cookie 2493 new ctx ffffff950581b968 cookie 2506
Jun 25 10:42:27 tici kernel: [ 202.797240] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 14 old ctx ffffff9505817128 cookie 2906 new ctx ffffff950581b968 cookie 2918
Jun 25 10:42:39 tici kernel: [ 214.597309] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 61 old ctx ffffff9505817128 cookie 3141 new ctx ffffff95058128e8 cookie 3154
Jun 25 10:42:40 tici kernel: [ 215.047413] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 16 old ctx ffffff95058128e8 cookie 3151 new ctx ffffff9505817128 cookie 3163
Jun 25 10:42:56 tici kernel: [ 231.397691] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 34 old ctx ffffff95058128e8 cookie 3477 new ctx ffffff95058128e8 cookie 3490
Jun 25 10:42:57 tici kernel: [ 232.447851] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 33 old ctx ffffff950581b968 cookie 3498 new ctx ffffff95058128e8 cookie 3511
Jun 25 10:43:21 tici kernel: [ 256.848371] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 6 old ctx ffffff95058128e8 cookie 3986 new ctx ffffff95058128e8 cookie 3999
Jun 25 10:43:35 tici kernel: [ 270.498221] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 48 old ctx ffffff950581b968 cookie 4259 new ctx ffffff95058128e8 cookie 4272
Jun 25 10:43:53 tici kernel: [ 288.898716] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 4 old ctx ffffff9505817128 cookie 4628 new ctx ffffff950581b968 cookie 4640
Jun 25 10:44:22 tici kernel: [ 317.049168] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 26 old ctx ffffff95058128e8 cookie 5191 new ctx ffffff950581b968 cookie 5203
Jun 25 10:44:25 tici kernel: [ 320.549074] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 53 old ctx ffffff9505817128 cookie 5260 new ctx ffffff9505817128 cookie 5273
Jun 25 10:44:35 tici kernel: [ 330.950070] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 11 old ctx ffffff950581b968 cookie 5468 new ctx ffffff9505817128 cookie 5481
Jun 25 10:44:47 tici kernel: [ 342.149429] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 9 old ctx ffffff9505817128 cookie 5692 new ctx ffffff95058128e8 cookie 5705
Jun 25 10:44:47 tici kernel: [ 342.349489] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 16 old ctx ffffff9505817128 cookie 5697 new ctx ffffff950581b968 cookie 5709
Jun 25 10:44:48 tici kernel: [ 343.599405] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 1 old ctx ffffff950581b968 cookie 5721 new ctx ffffff95058128e8 cookie 5734
Jun 25 10:44:54 tici kernel: [ 349.899540] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 59 old ctx ffffff9505817128 cookie 5847 new ctx ffffff9505817128 cookie 5860
Jun 25 10:45:02 tici kernel: [ 357.399664] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 24 old ctx ffffff950581b968 cookie 5997 new ctx ffffff95058128e8 cookie 6010
Jun 25 10:45:02 tici kernel: [ 357.699674] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 56 old ctx ffffff950581b968 cookie 6003 new ctx ffffff9505817128 cookie 6016
Jun 25 10:45:07 tici kernel: [ 362.299867] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 48 old ctx ffffff95058128e8 cookie 6096 new ctx ffffff950581b968 cookie 6108
Jun 25 10:45:07 tici kernel: [ 362.649761] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 21 old ctx ffffff950581b968 cookie 6102 new ctx ffffff9505817128 cookie 6115
Jun 25 10:45:09 tici kernel: [ 364.399893] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 34 old ctx ffffff9505817128 cookie 6137 new ctx ffffff95058128e8 cookie 6150
Jun 25 10:45:17 tici kernel: [ 372.749868] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 6 old ctx ffffff950581b968 cookie 6304 new ctx ffffff950581b968 cookie 6317
Jun 25 10:45:26 tici kernel: [ 381.050005] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 26 old ctx ffffff9505817128 cookie 6470 new ctx ffffff9505817128 cookie 6483
Jun 25 10:45:28 tici kernel: [ 383.200048] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 38 old ctx ffffff9505817128 cookie 6513 new ctx ffffff95058128e8 cookie 6526
Jun 25 10:45:43 tici kernel: [ 398.550639] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 19 old ctx ffffff950581b968 cookie 6820 new ctx ffffff95058128e8 cookie 6833
Since I don't have comma, here is the rlog and qlog
2022-06-25--08-39-59--114.zip
Provide a route where the issue occurs
2022-06-25--08-39-59--114
openpilot version
master-ci
Additional info
No response
Jun 25 10:40:01 tici kernel: [ 57.122581] CPU: 3 PID: 52540 Comm: dcamera.hevc Tainted: G C E 4.9.103+ #3
I think it's the dcamera, the crash sometimes happens when I open the driver view in settings without going on road
All of these warnings/error are seen during normal operation, and don't explain a device restart.
You want to look at the pstore section of the bootlogs, those contain the last kernel messages before the device rebooted and usually contain the actual crash dump.
But without your device uploading logs to the comma server I can't look into it.
Sorry for the inconvenience, if u could tell me where to look for the logs, I'll upload them.
Here are the bootlogs of the sessions bootlogs.zip
None of those contain a pstore entry. You can also find them in /sys/fs/pstore.
If you think there is something wrong with your hardware contact [email protected] and they can set up a warranty replacement.
I won't be able to get the pstore for the next 3 weeks as I'm out of town. I'll closing this for now as this is likely to be hardware issue. Will reopen if needed. Thanks
Reopening.
All of these warnings/error are seen during normal operation, and don't explain a device restart.
I don't think this is the case, I have another healthy C3 running the same code, and it doesn't have those lines. Also those lines repeat everytime the device reboots (attached syslog where it happens multiple times during 10 mins)
Excerpts from the log
Jul 22 11:43:12 tici kernel: [ 57.399915] i2c_geni 890000.i2c: i2c error :-107
...
Jul 22 11:43:13 tici kernel: [ 57.801122] CAM_INFO: CAM-CRM: cam_req_mgr_open: 130 cam_req_mgr_open:130 disalbe lpm
Jul 22 11:43:13 tici kernel: [ 57.801142] CAM_INFO: CAM-SENSOR: cam_pm_qos_update_request: 117 cam_pm_qos_update_request: update request 100
Jul 22 11:43:13 tici kernel: [ 57.918936] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 841 CAM_ACQUIRE_DEV Success, sensor_id:0x354,sensor_slave_addr:0x20
Jul 22 11:43:13 tici kernel: [ 57.988361] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 768 Probe success,slot:1,slave_addr:0x30,sensor_id:0x354
Jul 22 11:43:13 tici kernel: [ 57.994812] CAM_INFO: CAM: cam_res_mgr_shared_pinctrl_put: 273 Need hold put this pinctrl
Jul 22 11:43:13 tici kernel: [ 58.048188] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 841 CAM_ACQUIRE_DEV Success, sensor_id:0x354,sensor_slave_addr:0x30
Jul 22 11:43:13 tici kernel: [ 58.118340] CAM_INFO: CAM-SENSOR: cam_sensor_driver_cmd: 768 Probe success,slot:0,slave_addr:0x20,sensor_id:0x354
Jul 22 11:43:13 tici kernel: [ 58.124793] CAM_INFO: CAM: cam_res_mgr_shared_pinctrl_put: 273 Need hold put this pinctrl
Jul 22 11:43:14 tici kernel: [ 58.209991] msm_vidc: info: Opening video instance: 0000000000000000, 0
Jul 22 11:43:14 tici kernel: [ 58.214760] msm_vidc: info: Opening video instance: 0000000000000000, 0
Jul 22 11:43:14 tici kernel: [ 58.218366] msm_vidc: info: Opening video instance: 0000000000000000, 0
Jul 22 11:43:14 tici kernel: [ 58.248530] msm_vidc: info: Opening video instance: 0000000000000000, 0
...
Jul 22 11:43:14 tici kernel: [ 58.479416] PC is at vb2_warn_zero_bytesused.isra.10.part.11+0x24/0x68
Jul 22 11:43:14 tici kernel: [ 58.479425] LR is at __fill_vb2_buffer+0x4d0/0x508
Jul 22 11:43:14 tici kernel: [ 58.479434] pc : [<ffffff891ca990bc>] lr : [<ffffff891ca995d0>] pstate: 20400145
Jul 22 11:43:14 tici kernel: [ 58.479442] sp : ffffffd6cec2f7e0
Jul 22 11:43:14 tici kernel: [ 58.479450] x29: ffffffd6cec2f7e0 x28: ffffff891efcbbd8
Jul 22 11:43:14 tici kernel: [ 58.479472] x27: 0000000000000030 x26: ffffffd70dc15000
Jul 22 11:43:14 tici kernel: [ 58.479494] x25: ffffffd6cec2f8e8 x24: ffffffd70dc15000
Jul 22 11:43:14 tici kernel: [ 58.479516] x23: 0000000000000000 x22: ffffffd6cec2f8e8
Jul 22 11:43:14 tici kernel: [ 58.479538] x21: 0000000000000000 x20: ffffffd72f1f8000
Jul 22 11:43:14 tici kernel: [ 58.479560] x19: ffffffd72f1f8000 x18: 0000000000000000
Jul 22 11:43:14 tici kernel: [ 58.479582] x17: 0000007f9a666930 x16: ffffff891c2552d0
Jul 22 11:43:14 tici kernel: [ 58.479603] x15: 000056790b54c7a8 x14: 0000000000000000
Jul 22 11:43:14 tici kernel: [ 58.479625] x13: 0000000000000000 x12: 0000000000000000
Jul 22 11:43:14 tici kernel: [ 58.479647] x11: 0000000000000000 x10: 0000000000000000
Jul 22 11:43:14 tici kernel: [ 58.479668] x9 : 0000000000000000 x8 : ffffffd6cec2f918
Jul 22 11:43:14 tici kernel: [ 58.479690] x7 : 0000000000000000 x6 : 000000000000003f
Jul 22 11:43:14 tici kernel: [ 58.479712] x5 : 0000000000000000 x4 : 0000000000000030
Jul 22 11:43:14 tici kernel: [ 58.479733] x3 : 0000000000000000 x2 : 0000000000000008
Jul 22 11:43:14 tici kernel: [ 58.479755] x1 : 0000000000000001 x0 : ffffff891efcb000
Jul 22 11:43:14 tici kernel: [ 58.479778]
Jul 22 11:43:14 tici kernel: [ 58.479778] PC: 0xffffff891ca9907c:
Jul 22 11:43:14 tici kernel: [ 58.479785] 907c a8c27bfd d65f03c0 2a1403e1 97ffffce a94153f3 a8c27bfd d65f03c0 a9be7bfd
Jul 22 11:43:14 tici kernel: [ 58.479857] 909c 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f d0012980 52800021 392f7001
Jul 22 11:43:14 tici kernel: [ 58.479927] 90bc d4210000 b0006e80 91000000 97dcee68 f9400260 39406000 361000e0 b0006e80
Jul 22 11:43:14 tici kernel: [ 58.479997] 90dc 91014000 97dcee62 f9400bf3 a8c27bfd d65f03c0 b0006e80 91022000 97dcee5c
Jul 22 11:43:14 tici kernel: [ 58.480068]
Jul 22 11:43:14 tici kernel: [ 58.480068] LR: 0xffffff891ca99590:
Jul 22 11:43:14 tici kernel: [ 58.480076] 9590 b9403e60 17ffff3a b9400a60 35ffe480 b9401b20 b9001720 17ffff22 12802701
Jul 22 11:43:14 tici kernel: [ 58.480146] 95b0 52800015 0a010000 b9022280 17ffff7a b900173f 17ffff59 aa1403e0 97fffeb3
Jul 22 11:43:14 tici kernel: [ 58.480216] 95d0 f9400280 39406000 3617f100 b8776b00 17ffff49 aa1403e0 97fffeac f9400280
Jul 22 11:43:14 tici kernel: [ 58.480286] 95f0 39406000 3617fd20 b9400a60 b9001720 17ffff0c 00000000 a9be7bfd 910003fd
Jul 22 11:43:14 tici kernel: [ 58.480357]
Jul 22 11:43:14 tici kernel: [ 58.480357] SP: 0xffffffd6cec2f7a0:
Jul 22 11:43:14 tici kernel: [ 58.480364] f7a0 1ca995d0 ffffff89 cec2f7e0 ffffffd6 1ca990bc ffffff89 20400145 00000000
Jul 22 11:43:14 tici kernel: [ 58.480434] f7c0 2c83fe00 ffffffd7 00000096 00000000 ffffffff 0000007f 18eff120 ffffff80
Jul 22 11:43:14 tici kernel: [ 58.480504] f7e0 cec2f800 ffffffd6 1ca995d0 ffffff89 cec2fcf8 ffffffd6 1c1b29fc ffffff89
Jul 22 11:43:14 tici kernel: [ 58.480574] f800 cec2f860 ffffffd6 1ca92df4 ffffff89 00000001 00000000 2f1f8000 ffffffd7
Jul 22 11:43:14 tici kernel: [ 58.480645] ---[ end trace ff392848c91224f2 ]---
Jul 22 11:43:14 tici kernel: [ 58.480654] Call trace:
Jul 22 11:43:14 tici kernel: [ 58.480664] Exception stack(0xffffffd6cec2f5e0 to 0xffffffd6cec2f710)
Jul 22 11:43:14 tici kernel: [ 58.480675] f5e0: ffffffd72f1f8000 0000007fffffffff ffffffd6cec2f7e0 ffffff891ca990bc
Jul 22 11:43:14 tici kernel: [ 58.480684] f600: 0000000020400145 000000000000003d ffffffd6cec2f8e8 0000004e1fdce000
Jul 22 11:43:14 tici kernel: [ 58.480694] f620: ffffffd6cec2f650 ffffff891c08ebbc ffffff891e488788 ffffff891d24a024
Jul 22 11:43:14 tici kernel: [ 58.480704] f640: ffffff891d24a024 0000000000000446 ffffffd6cec2f6b0 ffffff891c0e3c10
Jul 22 11:43:14 tici kernel: [ 58.480713] f660: 0000000000000001 ffffff891c14cee4 ffffffd6cec2f6b0 ffffff8900000000
Jul 22 11:43:14 tici kernel: [ 58.480723] f680: ffffff891c13b76c ffffffd6cec2f730 ffffffd6cec2f720 ffffff891c13b76c
Jul 22 11:43:14 tici kernel: [ 58.480732] f6a0: ffffff89ffffffff e75aaa2ba134fa00 ffffff891efcb000 0000000000000001
Jul 22 11:43:14 tici kernel: [ 58.480742] f6c0: 0000000000000008 0000000000000000 0000000000000030 0000000000000000
Jul 22 11:43:14 tici kernel: [ 58.480751] f6e0: 000000000000003f 0000000000000000 ffffffd6cec2f918 0000000000000000
Jul 22 11:43:14 tici kernel: [ 58.480760] f700: 0000000000000000 0000000000000000
Jul 22 11:43:14 tici kernel: [ 58.480770] [<ffffff891ca990bc>] vb2_warn_zero_bytesused.isra.10.part.11+0x24/0x68
Jul 22 11:43:14 tici kernel: [ 58.480780] [<ffffff891ca995d0>] __fill_vb2_buffer+0x4d0/0x508
Jul 22 11:43:14 tici kernel: [ 58.480790] [<ffffff891ca92df4>] __qbuf_userptr+0x384/0x6d0
Jul 22 11:43:14 tici kernel: [ 58.480800] [<ffffff891ca94fb8>] __buf_prepare+0x118/0x208
Jul 22 11:43:14 tici kernel: [ 58.480810] [<ffffff891ca951f8>] vb2_core_qbuf+0x78/0x2a8
Jul 22 11:43:14 tici kernel: [ 58.480819] [<ffffff891ca98d00>] vb2_qbuf+0x68/0xb0
Jul 22 11:43:14 tici kernel: [ 58.480831] [<ffffff891cacb534>] msm_vidc_qbuf+0x17c/0x2f8
Jul 22 11:43:14 tici kernel: [ 58.480841] [<ffffff891cab1ae8>] msm_v4l2_qbuf+0x38/0x48
Jul 22 11:43:14 tici kernel: [ 58.480852] [<ffffff891ca7c8d4>] v4l_qbuf+0x4c/0x60
Jul 22 11:43:14 tici kernel: [ 58.480862] [<ffffff891ca7ba00>] __video_do_ioctl+0x210/0x2d0
Jul 22 11:43:14 tici kernel: [ 58.480871] [<ffffff891ca7b5bc>] video_usercopy+0x43c/0x620
Jul 22 11:43:14 tici kernel: [ 58.480880] [<ffffff891ca7b7dc>] video_ioctl2+0x3c/0x50
Jul 22 11:43:14 tici kernel: [ 58.480892] [<ffffff891ca7626c>] v4l2_ioctl+0xbc/0x108
Jul 22 11:43:14 tici kernel: [ 58.480904] [<ffffff891c254b84>] do_vfs_ioctl+0xc4/0x810
Jul 22 11:43:14 tici kernel: [ 58.480913] [<ffffff891c255360>] SyS_ioctl+0x90/0xa8
Jul 22 11:43:14 tici kernel: [ 58.480926] [<ffffff891c083f80>] el0_svc_naked+0x34/0x38
Jul 22 11:43:14 tici kernel: [ 58.480936] use of bytesused == 0 is deprecated and will be removed in the future,
Jul 22 11:43:14 tici kernel: [ 58.480948] use the actual size instead.
my device is banned (ec07911a91f863f3) for unknown reason, I checked pstore there is nothing there. But if u want to the device to uploads stuff again, pls unban it, at least for a bit so I have the chance to upload those files.
- Replaced NVME with a new one
- Took it to the office and
STARTED=1 ./scripts/launch_corolla.sh
Same behavior. It reboots after a while.
In these screenshots, the device rebooted 58 secs ago
tail -f kernlog
glances and the tmux session
Some interesting kern.log lines:
Jul 31 14:54:34 tici kernel: [ 1855.332660] i2c_geni 890000.i2c: i2c error :-107
Jul 31 14:54:35 tici kernel: [ 1856.556215] msm_vidc: err: got buffer back with error ea5e8
Jul 31 14:54:42 tici kernel: [ 1863.716893] CAM_ERR: CAM-CDM: cam_hw_cdm_submit_gen_irq: 364 cdm test remove bl tag 48 old ctx ffffff953da201a8 cookie 140 new ctx ffffff953da29228 cookie 153
My device has become so unstable that it's unwise to go on road with it. However I can reproduce the issue 100% of the time with just
STARTED=1 ./scripts/launch_corolla.sh
Route: ec07911a91f863f3%7C2022-08-17--14-59-31
