openpilot
openpilot copied to clipboard
commaai
GM: ACC faults
Events while engaged on 0.8.13-release
CHEVROLET VOLT PREMIER 2017 85a49685933befe2|2022-02-24--15-22-05--5 7
CHEVROLET VOLT PREMIER 2017 1f0e64fd786bf5ed|2022-02-26--15-42-18--1 6
CHEVROLET VOLT PREMIER 2017 e2b25f39da040caa|2022-02-26--16-38-50--10 7
CHEVROLET VOLT PREMIER 2017 1f0e64fd786bf5ed|2022-03-01--14-09-07--2 6
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-03-01--14-42-51--35 6
CADILLAC ESCALADE ESV 2016 495552ac53209990|2022-03-02--17-15-36--34 7
CADILLAC ESCALADE ESV 2016 495552ac53209990|2022-03-02--18-44-06--1 7
CHEVROLET VOLT PREMIER 2017 d2ccdcf33e62cfa0|2022-03-04--13-58-06--8 7
CHEVROLET VOLT PREMIER 2017 3954275d7161f9a8|2022-03-05--11-36-56--17 6
CADILLAC ESCALADE ESV 2016 2e5410d8378a30c6|2022-03-08--18-09-17--20 6
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-03-08--22-47-34--20 6
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-03-10--00-25-40--1 6
CADILLAC ESCALADE ESV 2016 495552ac53209990|2022-03-10--08-38-12--26 7
CADILLAC ESCALADE ESV 2016 495552ac53209990|2022-03-10--08-38-12--27 7
CADILLAC ESCALADE ESV 2016 495552ac53209990|2022-03-10--08-38-12--28 7
CADILLAC ESCALADE ESV 2016 495552ac53209990|2022-03-10--08-38-12--29 7
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-03-11--13-25-56--4 6
CHEVROLET VOLT PREMIER 2017 d2ccdcf33e62cfa0|2022-03-12--19-23-43--2 7
CHEVROLET VOLT PREMIER 2017 9425b0775a4688a1|2022-03-12--11-42-35--74 6
CHEVROLET VOLT PREMIER 2017 87b41dfea45f3cce|2022-03-12--13-49-33--96 6
CADILLAC ESCALADE ESV 2016 495552ac53209990|2022-03-13--18-36-37--9 7
CADILLAC ESCALADE ESV 2016 495552ac53209990|2022-03-13--18-54-37--12 7
CHEVROLET VOLT PREMIER 2017 d2ccdcf33e62cfa0|2022-03-13--18-50-46--3 7
CADILLAC ESCALADE ESV 2016 87b41dfea45f3cce|2022-03-16--19-32-51--4 6
CHEVROLET VOLT PREMIER 2017 87b41dfea45f3cce|2022-03-17--14-26-47--12 6
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-03-18--22-16-59--2 6
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-03-18--22-16-59--26 6
CHEVROLET VOLT PREMIER 2017 bfe8da33a68f85f2|2022-03-18--20-18-29--13 6
GMC ACADIA DENALI 2018 2cb99e3ec962642f|2022-03-19--14-13-59--7 7
CHEVROLET VOLT PREMIER 2017 bfe8da33a68f85f2|2022-03-19--13-02-59--31 6
CHEVROLET VOLT PREMIER 2017 a1e78442f3848534|2022-03-20--14-08-49--13 6
CHEVROLET VOLT PREMIER 2017 9425b0775a4688a1|2022-03-20--10-25-05--64 6
CHEVROLET VOLT PREMIER 2017 bfe8da33a68f85f2|2022-03-21--12-45-46--23 6
CHEVROLET VOLT PREMIER 2017 bfe8da33a68f85f2|2022-03-21--13-18-01--55 6
CHEVROLET VOLT PREMIER 2017 e2b25f39da040caa|2022-03-22--16-22-33--50 7
CHEVROLET VOLT PREMIER 2017 e2b25f39da040caa|2022-03-22--16-22-33--79 7
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-03-23--14-18-49--5 6
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-03-24--22-50-30--65 6
GMC ACADIA DENALI 2018 2cb99e3ec962642f|2022-03-27--11-17-29--11 7
GMC ACADIA DENALI 2018 2cb99e3ec962642f|2022-03-27--21-01-44--138 7
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-03-30--21-51-40--37 6
CHEVROLET VOLT PREMIER 2017 e2b25f39da040caa|2022-04-01--12-23-47--1 7
CHEVROLET VOLT PREMIER 2017 e2b25f39da040caa|2022-04-01--13-03-07--3 7
CHEVROLET VOLT PREMIER 2017 139dc5d40eb1be7c|2022-04-02--18-11-55--21 7
GMC ACADIA DENALI 2018 2cb99e3ec962642f|2022-04-02--18-45-14--9 7
CHEVROLET VOLT PREMIER 2017 bfe8da33a68f85f2|2022-04-03--19-18-53--8 6
CHEVROLET VOLT PREMIER 2017 590fd6ee1f594beb|2022-04-04--07-56-57--8 6
CHEVROLET VOLT PREMIER 2017 590fd6ee1f594beb|2022-04-05--07-50-52--6 6
CHEVROLET VOLT PREMIER 2017 a3e8ad84037083ca|2022-04-05--14-40-54--7 7
CHEVROLET VOLT PREMIER 2017 20cb801701fc8237|2022-04-05--20-35-08--2 7
CHEVROLET VOLT PREMIER 2017 20cb801701fc8237|2022-04-05--20-35-08--3 7
CHEVROLET VOLT PREMIER 2017 20cb801701fc8237|2022-04-05--20-35-08--5 7
CHEVROLET VOLT PREMIER 2017 20cb801701fc8237|2022-04-05--20-42-32--1 7
CHEVROLET VOLT PREMIER 2017 20cb801701fc8237|2022-04-06--00-25-59--3 7
CHEVROLET VOLT PREMIER 2017 20cb801701fc8237|2022-04-06--00-25-59--9 7
CHEVROLET VOLT PREMIER 2017 20cb801701fc8237|2022-04-06--00-25-59--11 7
CHEVROLET VOLT PREMIER 2017 a3e8ad84037083ca|2022-04-06--08-02-45--2 7
CHEVROLET VOLT PREMIER 2017 20cb801701fc8237|2022-04-06--13-09-20--8 7
CHEVROLET VOLT PREMIER 2017 2e5410d8378a30c6|2022-04-06--20-52-23--29 6
CHEVROLET VOLT PREMIER 2017 3954275d7161f9a8|2022-04-06--13-04-46--3 6
CHEVROLET VOLT PREMIER 2017 d2ccdcf33e62cfa0|2022-04-07--12-44-16--4 7
CHEVROLET VOLT PREMIER 2017 e2b25f39da040caa|2022-04-08--18-19-25--4 7
CHEVROLET VOLT PREMIER 2017 87b41dfea45f3cce|2022-04-11--07-45-46--5 6
CHEVROLET VOLT PREMIER 2017 87b41dfea45f3cce|2022-04-11--07-45-46--8 6
CHEVROLET VOLT PREMIER 2017 e2b25f39da040caa|2022-04-12--09-26-50--206 7
CHEVROLET VOLT PREMIER 2017 e2b25f39da040caa|2022-04-12--15-55-15--51 7
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-13--08-36-57--4 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-13--08-36-57--5 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-13--08-36-57--19 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-13--08-36-57--20 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-13--08-36-57--30 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-13--08-36-57--37 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-13--12-29-10--5 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-13--12-29-10--17 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-13--14-49-09--8 6
CHEVROLET VOLT PREMIER 2017 e2b25f39da040caa|2022-04-13--16-11-40--38 7
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-14--08-29-47--15 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-14--08-29-47--16 6
CHEVROLET VOLT PREMIER 2017 6363a0e48f515f27|2022-04-11--21-09-30--3 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-14--14-24-56--6 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-14--14-24-56--11 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-14--15-28-48--12 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-14--15-28-48--16 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-14--15-28-48--21 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-14--16-50-16--4 6
CHEVROLET VOLT PREMIER 2017 3e336a3be4261460|2022-04-14--18-23-56--35 6
CADILLAC ESCALADE ESV 2016 3e336a3be4261460|2022-04-15--09-57-35--4 6
CADILLAC ESCALADE ESV 2016 3e336a3be4261460|2022-04-15--09-57-35--17 6
Diagnosed my Volt ACC faults to a known brake-pressed-sensor vehicle issue.
- Symptoms
- Cruise fault on engage, "repeatedly fails to engage"
- Cruise fault on road bumps
- Cruise fault on tapping brakes
Narrative: vehicle ACC and openpilot are using different brake-pressed signals. Vehicle is more sensitive, and wants to disengage but openpilot continues with ACC commands. In 0.5 seconds, vehicle ACC faults.
- TODO
- [x] Find the correct ACC brake-pressed disengage signal
- [x] Brake pedal failure behavior should be: soft-disable "Brake pedal pressed"
- [x] Clean up documentation on this vehicle brake pedal failure, users may experience unexpected brake pedal disengagements. Mitigation is that users are always ready to resume control.
New brake pressed from same message as brake pedal position, will backtest.
I have been getting intermittent RSOD CAN errors when I press the brake when OpenPilot is engaged, but they resolve immediately. This just started happening, maybe since the GM brake signal was changed.
This PR is causing ACC faults on cars that have a noisy brake pedal: https://github.com/commaai/openpilot/pull/23712
The car's ACC system actually uses the noisy ECMAcceleratorPos->BrakePedalPos signal with a threshold of 8, and whenever it rises above that, the ACC system wants to disengage. openpilot knows the user didn't actually press the brake, so it stays engaged causing an ACC fault.
Ideally we should detect this and alert the user when we disengage to prevent a fault.
Looked into if there were any bits that describe that threshold, but there are none.
Plot showing a fault occurs shortly after that signal goes to 8, while the filtered/true brake signal (that we use now) stays zero:
https://github.com/qadmus/openpilot/tree/gm-fix-acc-fault-brake-position
Worked this morning with my light-brake-press test method.
Traveling this weekend, will make PR if no one gets to it first.
Looked at all the segments here that we have rlogs available for, most don't seem to be caused by any brake pedal issue:
Total segments with faults: 27, would be able to prevent by disengaging with brake pedal: 6
Reverted PR would have caught: 5 # https://github.com/commaai/openpilot/pull/23712
Master would have caught: 0
New PR would have caught: 6 # https://github.com/commaai/openpilot/pull/24589
Guys - the 8 threshold is not the same on all cars. The switch away from it originated when I needed to bump the threshold to 11 to accommodate trucks - The suburban, Silverado and Tahoe were getting random disengagements.
Adeeb requested finding and using a boolean signal, - both Jason Young and someone at comma confirmed the validity of the brake-pressed signal against lots of data.
@sshane - it was my understanding that the stock ACC is controlled by the ASCM, and that the ASCM must be disconnected for all currently supported GM vehicles. The ASCM sends gas and brake commands directly to the PT / Chassis busses.
OP completely replaces the functionality of the ASCM - where is this disengagement coming from?
We have an experimental VOACC branch that is able to control gas/brake without syncing any sort of ACC state. When the gas command is active, the car applies gas, etc.
Note PR #24589 is a new source of Brake Pedal Position, from ECMAcceleratorPos. The signals and subsequent thresholds are different than other brake position sources.
@JasonJShuler I'd guess the ECM disengages, as a redundant test of the brake pedal position. I don't trust the names of these messages though, don't know where they came from.
We may need BOTH (and more) brake pressed detections. And if models or worse, individual cars, have calibrated thresholds, which is possible, then finding a boolean signal is really valuable.
@JasonJShuler I'm not convinced all these faults are caused by the brake pedal issue, it's just a few could have been prevented and we wanted to revert back to a state where we would have caught at least some faults. If you believe different cars use different brake pressed signals/thresholds we should make a safety param flag for that based on the car. We should also figure out what is causing these faults in the process. I checked and blocked messages are also not the cause.
Yeah there must be a redundant check in the ECM (/PCM, or is that the ASCM?) where it won't allow you apply brake or gas if the user is pressing the brake, which it thinks they are based on the noisy signal we removed.
Guys - the 8 threshold is not the same on all cars. The switch away from it originated when I needed to bump the threshold to 11 to accommodate trucks - The suburban, Silverado and Tahoe were getting random disengagements.
We simply reverted back to avoid regressions for currently supported cars. If these new cars already kind of work, it'll be easier to find a solution if you open a PR and include some routes.
Quickly estimated how many of these faults would be fixed by https://github.com/commaai/openpilot/pull/25402. Seems like a pretty sizable chunk of the segments with rlogs. I can likely adapt to work on qlogs later.
The criteria was engaged shorter than 600 ms ago with the resume button (since we can be in any part of the route we don't know if we ever engaged after main toggle before this (can be solved with looking at the whole qlog route)):
Total segments with faults: 27
Likely caused by resume bug: 13
Camera ACC ACC faults:
- [x] Enabling with resume button held down for longer than x seconds causes fault every time
- [x] Enabling at a stop with the brake pressure between the thresholds of the camera and the ECM https://github.com/commaai/openpilot/pull/26079
- [x] Pressing cancel while stock system is braking -- thinks you've accidentally pressed the cancel and continues to brake for you. Something is sent to the ECM from the camera making it fault (not on CAN, must be GMLAN or FlexRay). Fixed by delaying our cancel until the camera does it for us (higher brake threshold) https://github.com/commaai/openpilot/pull/26081 (REAL FIX: https://github.com/commaai/openpilot/pull/26400 and https://github.com/commaai/openpilot/pull/26197)
There's still a lot I don't understand from this list. I may have figured out a sizable chunk of the remaining camera ACC faults, so I'll re-check all the ones above with rlogs
Bolt EUV camera ACC faults:
- [ ] Enabling at a stop shortly after reversing -- the first enable will result in an ACC fault
- [x] After rebooting the device while car is on (you can do this without an LKAS fault once or twice), we lost brake control and the car kept coasting into the lead:
6e997623cb7cdaa6|2023-02-09--16-19-40--0We check the ECM for cruise faults, but I guess the brake controller must have its own signal somewhere - [ ] Unknown ACC fault in the Bolt:
6e997623cb7cdaa6|2023-02-09--16-20-24--1