openpilot
openpilot copied to clipboard
commaai
[$200 bounty] [Ford Q4] Ford Lightnings Harness Causes Errors When Openpilot Not Running
Describe the bug
There's currently an issue with F150 Lightnings on the Ford Q4 Harness/CAN-FD platform. This is specific to the Lightning as the ICE variants do not have this issue nor do the Mustang Mach Es.
- Install Ford Q4 Harness into F150 Lightning.
- Turn on Vehicle and notice no errors
- Install Comma Power and plug into harness
- Turn on vehicle and notice no errors
- Plug in Comma provided USB-C cable to harness (leaving unplugged from Comma 3/3X)
- Notice the number of errors received (Traction Control, Park Assist, One Pedal Drive)
- Plug in Comma 3X (with a working release of OpenPilot) and notice issue persist.
- Allow Openpilot to fully boot and open the relay on the harness and observe the car faults disappearing
Which car does this affect?
2022+ Ford F150 Lightning
Provide a route where the issue occurs
N/A - Happens prior to openpilot
openpilot version
0.9.5
Additional info
I've spoken with Discord and they're pretty confused on why this is happening as well. Additionally, there's now 2 Lightnings that are observing this behavior -- mine and one other persons.
I've also emailed support and they were a bit confused as well.
Need a route to look into this. Does it also happen with openpilot toggled off in the settings?
I've responded in PM on discord due to sensitive info on the route. If that's not sufficient, I can get the info away from any sensitive locations.
Also experiencing this issue, let me know if I can provide any data. I can replicate with the exact steps as above. What really baffles me is the issue occurs without comma connected, like the harness box is immediately engaging relays when cable gets plugged in.
Something I tried yesterday that I haven't tried previously was plugging in the USB-C without anything else comma related plugged in.
So, I plugged in the harness. Then just immediately plugged in an OBD-C cable (provided by comma) and it still faults (even without the Ethernet plugged in from the Comma Power)
$200 bounty for root causing and coming up with a fix we can ship.
Is there any documentation on the wire harness box that I can access in order to troubleshoot? I've got some oscilloscopes/etc and can log some data.
All the harnesses are here: https://github.com/commaai/neo/tree/master/car_harness
It's worth noting that JUST the harness with an OBD-C cable is necessary to trigger faults.
So harness with relay box is okay, add the cable only and it fails.
Just installed Comma3X with the Ford Q4 harness in my Lightning - same behavior.
Curious if this is bluecruise or copilot assist related. Are both users xlt or lariat/platinum?
Has anyone tried looping pt1 - pt4 from connector to connector and avoiding the harness box. It’s a long shot but depending on what voltages are going through those and how long your obd c cable is there could be some weird electro magnetic stuff going on. At a minimum you would be removing variables from the situation.
Has anyone tried looping pt1 - pt4 from connector to connector and avoiding the harness box. It’s a long shot but depending on what voltages are going through those and how long your obd c cable is there could be some weird electro magnetic stuff going on. At a minimum you would be removing variables from the situation.
I don't think so. The cable is very short though. Using the one that comes from comma, you're looking at less than a foot of cable length.
I have taken measurements on some of the pins with a multimeter. I haven't been able to get CAN0H or CAN0L due to where the IPMA is and the length of wiring on the cables.
I was able to measure CAN1 H/L and they are sitting comfortably at 60ohms, which is what DebboR from Discord said it should be reading. However, I have seen CAN2 constantly jumping voltages. It eventually settles at 60ohms for a little bit before jumping all over the place again.
I would make sure the relay driving pins are isolated from any other pins on the harness box while the obd c cable is plugged in. Thinking through it more the pass through arent even on the obd c cable
I would make sure the relay driving pins are isolated from any other pins on the harness box while the obd c cable is plugged in. Thinking through it more the pass through arent even on the obd c cable
How would one do this? Is this something I can do being an idiot when it comes to hardware?
Having the harness box and obd c cable out of the car and with the cable plugged into the harness box, put one multimeter lead at the end of the obd c cable on SBU1 terminal and make sure all of the pins have on the harness box molex connector have no resistance/are open. Repeat on SBU2 https://github.com/commaai/neo/blob/master/car_harness/OBD-C.sch.pdf
Having the harness box and obd c cable out of the car and with the cable plugged into the harness box, put one multimeter lead at the end of the obd c cable on SBU1 terminal and make sure all of the pins have on the harness box molex connector have no resistance/are open. Repeat on SBU2 https://github.com/commaai/neo/blob/master/car_harness/OBD-C.sch.pdf
Does the OBD-C need to have power from something? Or can I literally just test on my desk for example
EDIT: Nvm, I think I see what you're saying.
It's a BlueCruise vehicle.
I have news. Maybe good, maybe bad. I'd loop DebboR but I'm not sure what his GH username is.
So, I've figured out why this was not seen on the external Red Panda. The external Red Panda didn't use a comma power to the harness. It was harness box -> red panda -> Comma 3.
The C3X allowed us to plug the Comma Power into the harness box. I've observed the following pattern:
- Comma approved OBD-C with no comma power -- no faults.
- Comma approved OBD-C with comma power -- faults
- Really long OBD-C cable -- faults.
- 1m OBD-C cable with no comma power -- no faults
I was able to test a ~6ft USB-C that sent the device to onroad but controlsd failed to start. Not sure about that.
What I think I've deduced is there's some kind of necessary shielding that's only possible on higher-end cables. Maybe some form of electromagnetic interference? From something on the vehicle side?
I'm working on getting a longer, higher quality OBD-C cable (maybe going to buy Apple's just for testing purposes).
Would love to hear if you guys have any suggestions but I do believe this is entirely hardware related and nothing we can ship in OP aside from maybe having the relay always open... which seems not ideal.
I'm not sure how comma wants to proceed. Once I find an OBD-C cable with enough length, I'm more than happy to link it. But currently on the lookout and trying to avoid the $200 cable.
The C3X allowed us to plug the Comma Power into the harness box. I've observed the following pattern:
- Comma approved OBD-C with no comma power -- no faults.
- Comma approved OBD-C with comma power -- faults
- Really long OBD-C cable -- faults.
- 1m OBD-C cable with no comma power -- no faults
Not a Comma or openpilot user, but my friend is that had this issue. I am an Electrical Engineer. Based on a brief read at what it seems Comma is doing here, which is sending the vehicles CAN bus down a long USB-C cable. This is either adding a unterminated leg to the CAN bus STAR configuration, assuming it already a star (which is not recommended in theory) or turning their flat bus into a star by adding that long leg and it's unterminated with the camera disconnected.
This would cause signal reflections on the CAN bus disrupting all the systems people are reporting. Remove the long leg of the star (created by the USB-C cable) reflections stop. Use a shorter cable, the reflections are less intense.
This would also explain the behavior you stated, assuming the Comma design doesn't have static termination but does so through some sort of CAN transceiver that has to be enabled by it's boot process.
This could be proved by creating a termination plug to go onto the end of the long USB-C cable when the camera is disconnected. It would have to be the USB connector and a 120 ohm resistor across the CAN data lines. I can't seems to find a pinout of the USB-C cable as wired by openpilot to see whats pins that would be though. Maybe someone can chime in on that?
As to why only the lightning? Well, EVs have much simpler CAN networks, usually only point to point or very few nodes while ICE vehicles are likely already a STAR configuration with many nodes, and as far as other EVs, well the Lightning is big and probably has the longest run of CAN bus already vs a MachE, which makes it more susceptible to reflection interference.
The C3X allowed us to plug the Comma Power into the harness box. I've observed the following pattern:
- Comma approved OBD-C with no comma power -- no faults.
- Comma approved OBD-C with comma power -- faults
- Really long OBD-C cable -- faults.
- 1m OBD-C cable with no comma power -- no faults
Not a Comma or openpilot user, but my friend is that had this issue. I am an Electrical Engineer. Based on a brief read at what it seems Comma is doing here, which is sending the vehicles CAN bus down a long USB-C cable. This is either adding a unterminated leg to the CAN bus STAR configuration, assuming it already a star (which is not recommended in theory) or turning their flat bus into a star by adding that long leg and it's unterminated with the camera disconnected.
This would cause single reflections on the CAN bus disrupting all the systems people are reporting. Remove the long leg of the star (created by the USB-C cable) reflections stop. Use a shorter cable, the reflections are less intense.
This would also explain the behavior you stated, assuming the Comma design doesn't have static termination but does so through some sort of CAN transceiver that has to be enabled by it's boot process.
This could be proved by creating a termination plug to go onto the end of the long USB-C cable when the camera is disconnected. It would have to be the USB connector and a 120 ohm resistor across the CAN data lines. I can't seems to find a pinout of the USB-C cable as wired by openpilot to see whats pins that would be though. Maybe someone can chime in on that?
As to why only the lightning? Well, EVs have much simpler CAN networks, usually only point to point or very few nodes while ICE vehicles are likely already a STAR configuration with many nodes, and as far as other EVs, well the Lightning is big and probably has the longest run of CAN bus already vs a MachE, which makes it more susceptible to reflection interference.
My smooth brain is having trouble following any of this.
https://github.com/commaai/neo/blob/master/car_harness/OBD-C.sch.pdf Does this help though?
My smooth brain is having trouble following any of this.
https://github.com/commaai/neo/blob/master/car_harness/OBD-C.sch.pdf Does this help though?
The final missing piece is how the relay box/harness is wired, as this would tell me what CAN busses are still passed through down the USB cable when the relay is "closed".
But worst case it's all, so a proper test would be to make a USB C female port (so the cable you have can plug into it at the camera side) and 4x 120 ohm resistors across the following pins:
A10-A11 A2-A3 B2-B3 B10-B11
If plugging in this "termination plug" in the problem cable fixes the issues with the vehicle, it proves my theory, though sadly i don't think there's a quick fix to make this mechanism act as a "Failsafe" as intended, because the camera coming unplugged can't magically make this shorting plug become connected, though maybe a passthrough shorting plug can be made, as it shouldn't hurt the Comma camera is the CAN buses are double terminated at the camera side, or even better be configured to not terminate the bus internally.
My smooth brain is having trouble following any of this. https://github.com/commaai/neo/blob/master/car_harness/OBD-C.sch.pdf Does this help though?
The final missing piece is how the relay box/harness is wired, as this would tell me what CAN busses are still passed through down the USB cable when the relay is "closed".
But worst case it's all, so a proper test would be to make a USB C female port (so the cable you have can plug into it at the camera side) and 4x 120 ohm resistors across the following pins:
A10-A11 A2-A3 B2-B3 B10-B11
If plugging in this "termination plug" in the problem cable fixes the issues with the vehicle, it proves my theory, though sadly i don't think there's a quick fix to make this mechanism act as a "Failsafe" as intended, because the camera coming unplugged can't magically make this shorting plug become connected, though maybe a passthrough shorting plug can be made, as it shouldn't hurt the Comma camera is the CAN buses are double terminated at the camera side, or even better be configured to not terminate the bus internally.
Agreed. I am looking at making a USB-C coupler out of two female USB-C breakout boards to support testing, I can add 120 ohm resistors to the coupler.
My smooth brain is having trouble following any of this.
https://github.com/commaai/neo/blob/master/car_harness/OBD-C.sch.pdf Does this help though?
The final missing piece is how the relay box/harness is wired, as this would tell me what CAN busses are still passed through down the USB cable when the relay is "closed".
But worst case it's all, so a proper test would be to make a USB C female port (so the cable you have can plug into it at the camera side) and 4x 120 ohm resistors across the following pins:
A10-A11 A2-A3 B2-B3 B10-B11
If plugging in this "termination plug" in the problem cable fixes the issues with the vehicle, it proves my theory, though sadly i don't think there's a quick fix to make this mechanism act as a "Failsafe" as intended, because the camera coming unplugged can't magically make this shorting plug become connected, though maybe a passthrough shorting plug can be made, as it shouldn't hurt the Comma camera is the CAN buses are double terminated at the camera side, or even better be configured to not terminate the bus internally.
Agreed. I am looking at making a USB-C coupler out of two female USB-C breakout boards to support testing, I can add 120 ohm resistors to the coupler.
Ditto. Just ordered two breakout boards.
I made the comment in discord about the relays in the harness box not disconnecting all can lines. From what little information there is in the schematic, I believe the relay only disconnects can0 and can2 then they send can1 and can3 down as a star.
I'm wondering if the harness box needs a second set of relays for that set of lines.
Two high level contributors brushed it off as a non hardware issue. If comma isn't using fail safe relays on all can bus communication across their "OBD-C" cable design, then that's a failure in safety in my opinion.
My smooth brain is having trouble following any of this. https://github.com/commaai/neo/blob/master/car_harness/OBD-C.sch.pdf Does this help though?
The final missing piece is how the relay box/harness is wired, as this would tell me what CAN busses are still passed through down the USB cable when the relay is "closed".
But worst case it's all, so a proper test would be to make a USB C female port (so the cable you have can plug into it at the camera side) and 4x 120 ohm resistors across the following pins:
A10-A11 A2-A3 B2-B3 B10-B11
If plugging in this "termination plug" in the problem cable fixes the issues with the vehicle, it proves my theory, though sadly i don't think there's a quick fix to make this mechanism act as a "Failsafe" as intended, because the camera coming unplugged can't magically make this shorting plug become connected, though maybe a passthrough shorting plug can be made, as it shouldn't hurt the Comma camera is the CAN buses are double terminated at the camera side, or even better be configured to not terminate the bus internally.
Im a computer engineer and I support this message.
I'm wondering if the harness box needs a second set of relays for that set of lines.
Two high level contributors brushed it off as a non hardware issue. If comma isn't using fail safe relays on all can bus communication across their "OBD-C" cable design, then that's a failure in safety in my opinion.
As someone who's has designed CAN bus systems before, I'm actually shocked this is how Openpilot was designed. I assumed looking at what my buddy told me, the "relay box" (which i never saw) was a box that had a MCU in it itself, to handle all the CAN bus tapping and "middle-man attacking" and it's logical controller just talked to the camera over the USB cable as a proper USB connection.
When i realized it's just some wire splices that add all these super long stubs to the CAN bus itself, all sorts of hairs on my neck started to tingle. Lots of design rules and approaches violated on something i would never cut any corners on (A motor vehicles communication bus).
I'm wondering if the harness box needs a second set of relays for that set of lines. Two high level contributors brushed it off as a non hardware issue. If comma isn't using fail safe relays on all can bus communication across their "OBD-C" cable design, then that's a failure in safety in my opinion.
As someone who's has designed CAN bus systems before, I'm actually shocked this is how Openpilot was designed. I assumed looking at what my buddy told me, the "relay box" (which i never saw) was a box that had a MCU in it itself, to handle all the CAN bus tapping and "middle-man attacking" and it's logical controller just talked to the camera over the USB cable as a proper USB connection.
When i realized it's just some wire splices that add all these super long stubs to the CAN bus itself, all sorts of hairs on my neck started to tingle. Lots of design rules and approaches violated on something i would never cut any corners on (A motor vehicles communication bus).
Probably relevant to note that Comma's design was built around the IPMA being behind the rear view mirror and the Comma on the windshield. Everything was 6 inches away from each other. It wasn't until Ford came along and put the IPMA under the dash (or in the trunk for some SUVs) that created issues with cable length.
Probably relevant to note that Comma's design was built around the IPMA being behind the rear view mirror and the Comma on the windshield. Everything was 6 inches away from each other. It wasn't until Ford came along and put the IPMA under the dash (or in the trunk for some SUVs) that created issues with cable length.
Thanks, that makes more sense. Basic rules for CAN bus design disallow Stub lengths longer than a critical length governed by bus speed. Even if terminated. and thats 1.7m and 3m for 500kbit and 250kbit respectively.
These USB cables with can bus signals... seem to be that long or worse.
Probably relevant to note that Comma's design was built around the IPMA being behind the rear view mirror and the Comma on the windshield. Everything was 6 inches away from each other. It wasn't until Ford came along and put the IPMA under the dash (or in the trunk for some SUVs) that created issues with cable length.
Thanks, that makes more sense. Basic rules for CAN bus design disallow Stub lengths longer than a critical length governed by bus speed. Even if terminated. and thats 1.7m and 3m for 500kbit and 250kbit respectively.
These USB cables with can bus signals... seem to be that long or worse.
It does seem that if long distances between the can device and the comma are going to become common, then the system needs a redesign so that the communications between the harness box and the Comma are true USB-C (or even ethernet), and the USB to CAN conversion happens at the harness box located right at the CAN device.