openpilot
openpilot copied to clipboard
commaai
GM camera ACC: LKAS fault when relay closes/opens
- [x] Fault when relay opens (we send counter of 1, should copy camera's counter) - https://github.com/commaai/openpilot/pull/26039
- [ ] Fault when relay opens with above fix - We can be sort of unlucky and try to send a message just before the relay opens, blocking it, then reading an updated counter from the camera (that doesn't reach the car), and end up skipping a counter. It's rare, but still a possibility.
- [ ] Fault when relay closes (our counter doesn't sync up with camera's)
0def4a390f6fe5c0|2022-10-25--19-17-40--35
Just had this happen today driving along straight and level near the end of the drive. Also had a little steering tune issue as well.
2022 EUV Premier non SC
I think this was the other time it happened.
0def4a390f6fe5c0|2022-10-22--14-22-47--23
I'm in a 2017 Bolt on an opgm/newcombo variant, but in case it helps your debugging, I had an LKAS fault while driving today on the segments:
82a0516f27e12844\|2022-11-04--18-03-04--15
82a0516f27e12844\|2022-11-04--18-03-04--16
@sshane we ran into this on our bolt euv while driving on a straight road (roughly 3 hours of drive time after we installed the comma 3) and it scared the crap out of us xD. Is there anything we can do to help fix this? We're pretty new to openpilot development so any advice would be appreciated! also we were on 0.9.0 stable in a 2022 bolt euv
Just happened to me today on the bolt-torque branch (0.9.1) on a 2022 Bolt EUV
Had another LKAS fault recently, my branch which is synced to comma master LKAS sync code.
I've had LKAS faults on this exact segment of road under similar conditions 3 or 4 times, leading me to think there's some reproducible issue there.
Its a long left hand sweeping on ramp with a meter light at the end. OP disengaged, heavy traffic, stop and go at 0-10 mph.
82a0516f27e12844|2022-12-08--17-07-17--22
It seems these errors are happening more often on the EUV. They do not always clear on a vehicle power cycle. I turn off the car, open then close the door, wait for the dash to go dark and then power on the car. The comma still says LKAS fault. Then I drive for a while as I have somewhere to go and don't care about the comma working or not and the cruise control won't work so I unplug the comma and there's a terribly annoying beeping and the cruise control works. This is very frustrating and makes me want to return the comma since at this point about 25% of my drives, it doesn't work. The LKAS error came up at some point when the vehicle was sitting charging tonight.
@ncoop23 you need to wait at least 3 minutes between closing the door and turning the bolt back on, seems to fix the error every time for me at least. This is cause the car doesn't completely shut off until after 3 minutes of being "off".
That's correct, I've observed 2-3 minutes with the car fully off to reset the fault. @ncoop23 when do these faults mostly happen for you? While driving? When first turning the car on? Or shortly after turning the car off?
@dominicdoty I've requested upload, can you put your device on wifi?
@abueide let me know if you have a segment where it happens and I can take a look
That's correct, I've observed 2-3 minutes with the car fully off to reset the fault. @ncoop23 when do these faults mostly happen for you? While driving? When first turning the car on? Or shortly after turning the car off?
I generally get them when driving for a while. Some have been near turns, other on the straight and level. I think there may be a buffer overflow or a command that needs an ack that doesn't get it like a calibration or check every so often. About to find and upload the latest one. Car is syncing. They are random. The strange thing is something latches as if I disconnect the comma there is a terribly annoying beeping coming from somewhere and I don't know if it's the comma.
@sshane i'll figure out how to use the segments system and keep my eye out for it happening again and post back here if I do. so far i had one single fault happen while driving (this was on stable 0.9.0 no experimental) in about 400 miles. we also had one that happened shortly after parking turning the car off, then realizing i needed to repark and turning it on again (not enough time for a full restart) it happened again.
@sshane Looks like the first one I posted I must have unpreserved, the one from 12/8 should have uploaded now - apologies for the delay.
Let me know if there's anything I can do to help out. These errors are pretty rare for me, but would be nice to get them fixed.
@sshane this is @abueide 's recent LKAS failure route ID: c9dc069b00b41849|2022-12-14--14-11-59--26. It happens right at the end so you may want c9dc069b00b41849|2022-12-14--14-11-59--27 as well.
another one, back to opgm fork, exact same location and conditions
82a0516f27e12844|2022-12-15--18-01-41--17
Happened to me again at 10:22 to 10:23am on Thursday dec 22nd.
6afbe9126b0b488e
I went to look for the video on connect but it’s not showing up. Maybe the logs still exist but I can see them. If it shows up I’ll post a proper link here.
Got one uploaded finally.
6afbe9126b0b488e|2023-01-02--12-35-07--2
Check around 12:37 that’s when it triggered.
I set the flag on connect to preserve this drive.
And in case I forgot to mention it I drive a 2022 Bolt EUV Premiere (No SC). Running on master-ci branch.
So what happens to correct this issue? It's kind of a big deal since it causes the comma to abort and need a car restart.
I've been busy with other work recently, but will look at all the issues. Make sure if you report a segment to upload all logs and preserve
@minorOffense I can't guarantee this will work, it's just a hypothesis I have, but can you try this branch?
https://github.com/commaai/openpilot/tree/gm-full-rate-fault Installer URL: https://installer.comma.ai/commaai/gm-full-rate-fault
The two faults you posted the car had a temporary fault followed by a permanent fault, it could be possible that openpilot slowing its send of the inactive steer frame could be exacerbating the issue during the temp fault, leading to a perm fault.
Please drive on it for a few days and let me know with a route if you get any more temporary faults, and if it recovers or not.
@minorOffense I can't guarantee this will work, it's just a hypothesis I have, but can you try this branch?
https://github.com/commaai/openpilot/tree/gm-full-rate-fault Installer URL: https://installer.comma.ai/commaai/gm-full-rate-fault
The two faults you posted the car had a temporary fault followed by a permanent fault, it could be possible that openpilot slowing its send of the inactive steer frame could be exacerbating the issue during the temp fault, leading to a perm fault.
Please drive on it for a few days and let me know with a route if you get any more temporary faults, and if it recovers or not.
I'm driving around on this now. So far so good. The steering tuning isn't damped enough but that's something else.
Had what looked like 2 steering errors in quick succession on your test build yesterday. Nothing caused comma to fault, only warn. 0def4a390f6fe5c0|2023-01-20--21-51-56--12
Please share if this is the actual error. I flagged a couple other things also.
@sshane I have been testing this change (applied to another branch of SunnyPilot) for a few days and thought the LKAS fault had been resolved, unfortunately just now I got the error TAKE CONTROL NOW! LKAS Fault - Restart car to engage.
48c21cb1105d4d1c|2023-01-23--19-30-58--5
Happened around 19:36:10. I was driving mostly straight at the time. 2022 Bolt EUV. Let me know if there's anything else I can do to help.
Got a take control now error yesterday on straight and level freeway with this branch. Same non SC EUV Premier
0def4a390f6fe5c0|2023-01-22--20-13-34--70
I’ve been stuck at home with COVID all week so I haven’t had a chance to test. Hopefully next week I’ll get a chance.
I have a branch now to reduce the send rate of the LKAS message to the EPS, as we suspect that sending two LKAS messages too close together is causing faults (undetectable by openpilot at just 100Hz). This branch changes the 50Hz message down to just 33Hz, so steering may be impacted, but if this hypothesis is correct, faulting should be reduced.
Let me know how this branch does @ncoop23 @jettoblack @minorOffense
Installer URL: https://installer.comma.ai/commaai/gm-reduce-rate
https://github.com/commaai/openpilot/tree/gm-reduce-rate
Are you able to see how many people are currently using a branch? Just curious.
I am installing the gm reduce rate branch and will report issues.
