openpilot LKAS Fault

Describe the bug

After the release of OpenPilot 0.9.5 I have been getting daily LKAS Faults on my 2023 Bolt EUV.

Which car does this affect?

2023 Bolt EUV

Provide a route where the issue occurs

I-5 Olympia to Highway 512 East and then to Puyallup.

openpilot version

0.9.5

Additional info

0.9.4 worked okay but it seems to occur for no reason on 0.9.5. Very unchill.

Dec 02 '23 01:12 Luft1950

What's your dongle ID? Do you have a route where this happens?

Dec 02 '23 01:12 adeebshihadeh

Hi Adeeb,

My dongle id is c93a84c2e28979fe. I travel from Olympia WA to the 512 cutoff then to Puyallup WA.

Thanks,.

From: "Adeeb Shihadeh" @.> To: "commaai/openpilot" @.> Cc: "Luft1950" @.>, "Author" @.> Sent: Friday, December 1, 2023 5:33:47 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

What's your dongle ID? Do you have a route where this happens?

— Reply to this email directly, [ https://github.com/commaai/openpilot/issues/30571#issuecomment-1836982973 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC24OH76PSWHAWMFI3PCKADYHKAPXAVCNFSM6AAAAABADUAJJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZWHE4DEOJXGM | unsubscribe ] . You are receiving this because you authored the thread. Message ID: @.***>

Dec 02 '23 02:12 Luft1950

Thanks, I'm looking into it. I found five events, but they're only from your car:

CHEVROLET BOLT EUV 2022                  c93a84c2e28979fe|2023-11-25--13-28-19--55
CHEVROLET BOLT EUV 2022                  c93a84c2e28979fe|2023-11-27--16-06-53--22
CHEVROLET BOLT EUV 2022                  c93a84c2e28979fe|2023-11-28--11-46-47--46
CHEVROLET BOLT EUV 2022                  c93a84c2e28979fe|2023-11-30--12-24-03--45
CHEVROLET BOLT EUV 2022                  c93a84c2e28979fe|2023-12-01--12-05-38--36

Dec 02 '23 03:12 adeebshihadeh

In the first case, right before the LKAS fault, I saw that an LKAS message/counter was missing on the 128 read back bus, so openpilot correctly re-sent the same counter, then it immediately faulted. Usually the EPS is robust to sending at a slower rate, it looked like the message might have actually got sent, but not added to bus 128 (thus duplicated counter)? pandaStates indicates no dropped messages though.

Dec 07 '23 19:12 sshane

Thanks Shane. I can't say that I understand a lot about what you said but it's good to know that you're on it. Any chance this will be fixed in the next version of OpenPilot? Thanks again.

From: "Shane Smiskol" @.> To: "commaai/openpilot" @.> Cc: "Luft1950" @.>, "Author" @.> Sent: Thursday, December 7, 2023 11:01:01 AM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

In the first cases, right before the LKAS fault, I saw that an LKAS message/counter was missing on the 128 read back bus, so openpilot correctly re-sent the same counter, then it immediately faulted. Usually the EPS is robust to sending at a slower rate, it looked like the message might have actually got sent, but not added to bus 128? pandaStates indicates no dropped messages though.

[ https://github.com/commaai/openpilot/assets/25857203/57d8b0f5-fc58-4fef-8aad-a5b041744e6c | image.png (view on web) ]

— Reply to this email directly, [ https://github.com/commaai/openpilot/issues/30571#issuecomment-1845938446 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC24OH3RK7R4632RCBJVM4LYIIG63AVCNFSM6AAAAABADUAJJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBVHEZTQNBUGY | unsubscribe ] . You are receiving this because you authored the thread. Message ID: @.***>

Dec 07 '23 20:12 Luft1950

For what it's worth I see the same issue on my 2022 Bolt EUV

Jan 31 '24 02:01 nharada1

I am also getting similar issue in Mazda 2024 every time during my 1st trip each day .

e548ddc194075bfb|2024-01-31--16-12-06--0

Feb 01 '24 20:02 siddchou

@Luft1950 does this still happen?

Mar 20 '24 00:03 sshane

I can let you know in about two weeks. It only happened when I drove from Tenino to Puyallup for my chemo treatments. I'll be starting my next cycle this coming Friday and will last till Thursday.

From: "Shane Smiskol" @.> To: "commaai/openpilot" @.> Cc: "Luft1950" @.>, "Mention" @.> Sent: Tuesday, March 19, 2024 5:53:57 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

[ https://github.com/Luft1950 | @Luft1950 ] does this still happen?

— Reply to this email directly, [ https://github.com/commaai/openpilot/issues/30571#issuecomment-2008472639 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC24OHYQAHKD6X2ZYCE3PMDYZDM2LAVCNFSM6AAAAABADUAJJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBYGQ3TENRTHE | unsubscribe ] . You are receiving this because you were mentioned. Message ID: @.***>

Mar 20 '24 02:03 Luft1950

Shane, This last week it did happen once. Tuesday March 26th on the first drive. (That video is about to be rotated off.) It was weird. Normally I get three "Take Control Immediately before it throws a fault but this time I think it only game me one.

From: "ERIC DAMRON" @.> To: "commaai" @.> Sent: Tuesday, March 19, 2024 7:03:32 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

I can let you know in about two weeks. It only happened when I drove from Tenino to Puyallup for my chemo treatments. I'll be starting my next cycle this coming Friday and will last till Thursday.

From: "Shane Smiskol" @.> To: "commaai/openpilot" @.> Cc: "Luft1950" @.>, "Mention" @.> Sent: Tuesday, March 19, 2024 5:53:57 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

[ https://github.com/Luft1950 | @Luft1950 ] does this still happen?

— Reply to this email directly, [ https://github.com/commaai/openpilot/issues/30571#issuecomment-2008472639 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC24OHYQAHKD6X2ZYCE3PMDYZDM2LAVCNFSM6AAAAABADUAJJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBYGQ3TENRTHE | unsubscribe ] . You are receiving this because you were mentioned. Message ID: @.***>

Mar 29 '24 15:03 Luft1950

Shane, the LKAS fault happened again just today. No "Take control" warnings at all. The video is for Monday April 8th 12:50 to 13:38.

From: "ERIC DAMRON" @.> To: "commaai" @.> Sent: Friday, March 29, 2024 8:00:14 AM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

Shane, This last week it did happen once. Tuesday March 26th on the first drive. (That video is about to be rotated off.) It was weird. Normally I get three "Take Control Immediately before it throws a fault but this time I think it only game me one.

From: "ERIC DAMRON" @.> To: "commaai" @.> Sent: Tuesday, March 19, 2024 7:03:32 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

I can let you know in about two weeks. It only happened when I drove from Tenino to Puyallup for my chemo treatments. I'll be starting my next cycle this coming Friday and will last till Thursday.

From: "Shane Smiskol" @.> To: "commaai/openpilot" @.> Cc: "Luft1950" @.>, "Mention" @.> Sent: Tuesday, March 19, 2024 5:53:57 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

[ https://github.com/Luft1950 | @Luft1950 ] does this still happen?

— Reply to this email directly, [ https://github.com/commaai/openpilot/issues/30571#issuecomment-2008472639 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC24OHYQAHKD6X2ZYCE3PMDYZDM2LAVCNFSM6AAAAABADUAJJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBYGQ3TENRTHE | unsubscribe ] . You are receiving this because you were mentioned. Message ID: @.***>

Apr 08 '24 21:04 Luft1950

I ended up simply returning my Comma, this issue was too prevalent to safely drive.

Apr 08 '24 21:04 nharada1

@Luft1950 I requested upload, please put the device on WiFi for 30 minutes to let the logs upload.

Apr 08 '24 21:04 sshane

Just got your request. I hope you were able to get the upload. The car is on WiFi now.

From: "Shane Smiskol" @.> To: "commaai/openpilot" @.> Cc: "Luft1950" @.>, "Mention" @.> Sent: Monday, April 8, 2024 2:17:37 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

[ https://github.com/Luft1950 | @Luft1950 ] I requested upload, please put the device on WiFi for 30 minutes to let the logs upload.

— Reply to this email directly, [ https://github.com/commaai/openpilot/issues/30571#issuecomment-2043657203 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC24OH5PHSUTNGG5HDGDQUTY4MCPDAVCNFSM6AAAAABADUAJJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBTGY2TOMRQGM | unsubscribe ] . You are receiving this because you were mentioned. Message ID: @.***>

Apr 09 '24 01:04 Luft1950

Looks like there is a problem with the panda reporting that the message was delivered. openpilot then re-sends the same counter as it thinks it wasn't delivered, but since it faulted here and there's no blocked or lost tx messages supposedly it did actually get sent.

Apr 09 '24 01:04 sshane

Shane, I don't know if you want me to tell you about each LKAS fault. If not just let me know. Today I got two. One when I was on my way to the cancer center and one when I went home. The videos are Sat. Apr 27 11:46 to 12:45 and Sat Apr 27 13:18 to 14:05. Thanks.

From: "Shane Smiskol" @.> To: "commaai/openpilot" @.> Cc: "Luft1950" @.>, "Mention" @.> Sent: Monday, April 8, 2024 6:53:51 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

Looks like there is a problem with the panda reporting that the message was delivered. openpilot then re-sends the same counter as it thinks it wasn't delivered, but since it faulted here and there's no blocked or lost tx messages supposedly it did actually get sent.

[ https://github.com/commaai/openpilot/assets/25857203/d512028c-774c-4ef9-beb9-a57b8683b5ee | image.png (view on web) ]

— Reply to this email directly, [ https://github.com/commaai/openpilot/issues/30571#issuecomment-2044001584 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC24OH2HM4RXKD55TLLFKILY4NC27AVCNFSM6AAAAABADUAJJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBUGAYDCNJYGQ | unsubscribe ] . You are receiving this because you were mentioned. Message ID: @.***>

Apr 28 '24 02:04 Luft1950

Shane, I got another LKAS fault today. The video is Sunday, April 28, 7:16am to 8:12 am.

Thanks.

From: "ERIC DAMRON" @.> To: "commaai" @.> Cc: "commaai" @.>, "Mention" @.> Sent: Saturday, April 27, 2024 7:02:20 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

Shane, I don't know if you want me to tell you about each LKAS fault. If not just let me know. Today I got two. One when I was on my way to the cancer center and one when I went home. The videos are Sat. Apr 27 11:46 to 12:45 and Sat Apr 27 13:18 to 14:05. Thanks.

From: "Shane Smiskol" @.> To: "commaai/openpilot" @.> Cc: "Luft1950" @.>, "Mention" @.> Sent: Monday, April 8, 2024 6:53:51 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

Looks like there is a problem with the panda reporting that the message was delivered. openpilot then re-sends the same counter as it thinks it wasn't delivered, but since it faulted here and there's no blocked or lost tx messages supposedly it did actually get sent.

[ https://github.com/commaai/openpilot/assets/25857203/d512028c-774c-4ef9-beb9-a57b8683b5ee | image.png (view on web) ]

— Reply to this email directly, [ https://github.com/commaai/openpilot/issues/30571#issuecomment-2044001584 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC24OH2HM4RXKD55TLLFKILY4NC27AVCNFSM6AAAAABADUAJJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBUGAYDCNJYGQ | unsubscribe ] . You are receiving this because you were mentioned. Message ID: @.***>

Apr 28 '24 22:04 Luft1950

Shane, I got another LKAS fault yesterday. It was a weird one. I was returning from my chemo and had received two "Take control immediately" warnings so I disengaged OpenPilot before the fault. I then drove without OpenPilot to Olympia. I looked up and the Comma 3X was displaying the LKAS fault! So my device faulted while being disengaged. The video is Monday April 29, 15:19 to 16:36. Thanks

From: "ERIC DAMRON" @.> To: "commaai" @.> Sent: Sunday, April 28, 2024 3:07:38 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

Shane, I got another LKAS fault today. The video is Sunday, April 28, 7:16am to 8:12 am.

Thanks.

From: "ERIC DAMRON" @.> To: "commaai" @.> Cc: "commaai" @.>, "Mention" @.> Sent: Saturday, April 27, 2024 7:02:20 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

Shane, I don't know if you want me to tell you about each LKAS fault. If not just let me know. Today I got two. One when I was on my way to the cancer center and one when I went home. The videos are Sat. Apr 27 11:46 to 12:45 and Sat Apr 27 13:18 to 14:05. Thanks.

From: "Shane Smiskol" @.> To: "commaai/openpilot" @.> Cc: "Luft1950" @.>, "Mention" @.> Sent: Monday, April 8, 2024 6:53:51 PM Subject: Re: [commaai/openpilot] LKAS Fault (Issue #30571)

Looks like there is a problem with the panda reporting that the message was delivered. openpilot then re-sends the same counter as it thinks it wasn't delivered, but since it faulted here and there's no blocked or lost tx messages supposedly it did actually get sent.

[ https://github.com/commaai/openpilot/assets/25857203/d512028c-774c-4ef9-beb9-a57b8683b5ee | image.png (view on web) ]

— Reply to this email directly, [ https://github.com/commaai/openpilot/issues/30571#issuecomment-2044001584 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/AC24OH2HM4RXKD55TLLFKILY4NC27AVCNFSM6AAAAABADUAJJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBUGAYDCNJYGQ | unsubscribe ] . You are receiving this because you were mentioned. Message ID: @.***>

Apr 30 '24 19:04 Luft1950

Can you go in connect and send the link for each?

Apr 30 '24 21:04 sshane

@Luft1950 You're the only one hitting this old issue. Currently this appears to be a software bug, but we don't currently have the time to root cause your specific case. We are going to allow you to return your device if you would like, just send this GitHub thread: https://shop.comma.ai/a/returns

Related to https://github.com/commaai/openpilot/issues/28899, the CAN packet dropped out for a few frames where we were expecting the loopback bus to say that the LKAS message was transmitted, so we transmitted it again incorrectly.

May 01 '24 23:05 sshane