raspberry-pi-pcie-devices
raspberry-pi-pcie-devices copied to clipboard
geerlingguy
Test the Time Appliances Project open Time Card
Look what arrived in the mail today...
It's Meta's open source hardware Time Card—read more about the card from the blog post Open-sourcing a more precise time appliance.
I'll be hooking it up to a CM4 and seeing what, if anything, it can do. The CM4's BCM54210PE PHY chip hardware has PTP support—I'll be attempting to see if I can get it to work with the Time Card to make the Pi be a viable option for a network's main time server. (None of the other Pi 4 models have PTP...).
With a fresh 64-bit Pi OS flash:
pi@cm4:~ $ uname -a
Linux cm4 5.10.17-v8+ #1414 SMP PREEMPT Fri Apr 30 13:23:25 BST 2021 aarch64 GNU/Linux
pi@cm4:~ $ sudo lspci -vvvv
...
01:00.0 Memory controller: Device 1d9b:0400
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Region 0: Memory at 600000000 (32-bit, non-prefetchable) [disabled] [size=32M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable- Count=1/16 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 1, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
First attempt, just yolo the driver into existence:
$ git clone https://github.com/opencomputeproject/Time-Appliance-Project.git
$ cd Time-Appliance-Project/Time-Card/DRV/
$ sudo apt-get install raspberrypi-kernel-headers # needed for source build
$ ./remake
make: Entering directory '/usr/src/linux-headers-5.10.52-v8+'
CC [M] /home/pi/Time-Appliance-Project/Time-Card/DRV/ptp_ocp.o
/home/pi/Time-Appliance-Project/Time-Card/DRV/ptp_ocp.c:956:22: warning: ‘struct firmware’ declared inside parameter list will not be visible outside of this definition or declaration
const struct firmware *fw)
^~~~~~~~
/home/pi/Time-Appliance-Project/Time-Card/DRV/ptp_ocp.c: In function ‘ptp_ocp_devlink_flash’:
/home/pi/Time-Appliance-Project/Time-Card/DRV/ptp_ocp.c:968:12: error: dereferencing pointer to incomplete type ‘const struct firmware’
resid = fw->size;
^~
/home/pi/Time-Appliance-Project/Time-Card/DRV/ptp_ocp.c: In function ‘ptp_ocp_devlink_flash_update’:
/home/pi/Time-Appliance-Project/Time-Card/DRV/ptp_ocp.c:1012:50: error: ‘struct devlink_flash_update_params’ has no member named ‘fw’
err = ptp_ocp_devlink_flash(devlink, dev, params->fw);
^~
make[1]: *** [scripts/Makefile.build:280: /home/pi/Time-Appliance-Project/Time-Card/DRV/ptp_ocp.o] Error 1
make: *** [Makefile:1824: /home/pi/Time-Appliance-Project/Time-Card/DRV] Error 2
make: Leaving directory '/usr/src/linux-headers-5.10.52-v8+'
I should note I also had to modify the remake script to detect Debian (right now it only works for Ubuntu and CentOS, explicitly). I opened an issue here: https://github.com/opencomputeproject/Time-Appliance-Project/issues/23
Option 2: Recompile 5.12+ kernel with PTP support... we'll see if I can get that working? From menuconfig on 5.14 kernel:
Symbol: PTP_1588_CLOCK_OCP [=n] │
│ Type : tristate │
│ Defined at drivers/ptp/Kconfig:156 │
│ Prompt: OpenCompute TimeCard as PTP clock │
│ Depends on: PTP_1588_CLOCK [=n] && HAS_IOMEM [=y] && PCI [=y] │
│ Location: │
│ -> Device Drivers │
│ (7) -> PTP clock support
Using my cross-compile environment, I set the following option in menuconfig:
Then compiled the kernel, copied it over, and rebooted and...
pi@cm4:~ $ dmesg | grep ptp
[ 4.425124] ptp_ocp 0000:01:00.0: enabling device (0000 -> 0002)
[ 4.425224] ptp_ocp 0000:01:00.0: Time: 3603.563280960, UNSYNCED
[ 4.427082] ptp_ocp 0000:01:00.0: Version 1.2.0, clock PPS, device ptp0
[ 4.427123] ptp_ocp 0000:01:00.0: TOD Version 2.0.1
[ 4.427143] ptp_ocp 0000:01:00.0: control: 10000001
[ 4.427156] ptp_ocp 0000:01:00.0: TOD Protocol UBX enabled
[ 4.427171] ptp_ocp 0000:01:00.0: GNSS ALL
[ 4.427187] ptp_ocp 0000:01:00.0: status: 0
[ 4.427202] ptp_ocp 0000:01:00.0: correction: 0
[ 4.427216] ptp_ocp 0000:01:00.0: utc_status: 0
[ 4.427229] ptp_ocp 0000:01:00.0: utc_offset: 0 valid:0 leap_valid:0
Yay!
The version of the driver you tried to compile seems to need kernel 5.11 or newer as far as I can tell by the error. Take a look here devlink.h in 5.11 vs devlink.h in 5.10.
@philhartung - Ah, thanks for that hint, seems like this could be annoying with Debian as I believe Bullseye even ships with 5.10 — will have to use a different OS or compile custom kernel to be able to use the driver.
We'll see how that comes along (see new comment just above).
Looks like the driver is working. Now checking out device tree:
$ ls /dev | grep ^p
...
ptp0
pts
Another thing I'd like to investigate relates to the Pi issue linked in the OP—basically, getting hardware timestamping working on eth0. Right now it's not:
$ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none
See: CM4 is missing IEEE1588-2008 support through BCM54210PE
Using ptp4l and phc2sys via sudo apt-get install -y linuxptp:
pi@cm4:~ $ sudo systemctl start ptp4l
pi@cm4:~ $ sudo systemctl status ptp4l
● ptp4l.service - Precision Time Protocol (PTP) service
Loaded: loaded (/lib/systemd/system/ptp4l.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2021-08-20 15:59:59 CDT; 29s ago
Docs: man:ptp4l
Process: 692 ExecStart=/usr/sbin/ptp4l -f /etc/linuxptp/ptp4l.conf -i eth0 (code=exited, status=255/EXCEPTION)
Main PID: 692 (code=exited, status=255/EXCEPTION)
Aug 20 15:59:59 cm4 systemd[1]: Started Precision Time Protocol (PTP) service.
Aug 20 15:59:59 cm4 ptp4l[692]: [238.440] interface 'eth0' does not support requested timestamping mode
Aug 20 15:59:59 cm4 ptp4l[692]: failed to create a clock
Aug 20 15:59:59 cm4 systemd[1]: ptp4l.service: Main process exited, code=exited, status=255/EXCEPTION
Aug 20 15:59:59 cm4 systemd[1]: ptp4l.service: Failed with result 'exit-code'.
So I'll get back to that later.
Going to have to pause for the day!
Just linking to one quick resource I was reading through for my future reference: Precision Time Protocol on Linux ~ Introduction to linuxptp.
I'm just gonna think out loud: ptp4l is not starting, because the default timestamping mode is hardware timestamping. As IEEE1588-2008 hardware timestamping is currently not supported by the driver for the BCM54210PE used in the CM4 (as indicated by the output of ethtool -T eth0), we'll have to wait for a driver supporting hardware timestamping. There is an incomplete implementation (RX working but TX is not, which defeats the purpose) that can be found here, but that is based on kernel version 5.10 where the driver for the Time Card didn't compile. It think it wouldn't be too hard to get the incomplete driver working under 5.11, but without TX timestamping that is a waste of time in my opinion.
One option to get the Time Card working with IEEE1588-2008 before a driver is ready is to switch to software timestamping in the ptp4l config. Software timestamping is quite inaccurate though and doesn't do the accuracy of the Time Card justice.
I have one further suggestion: If you get PTP working on the CM4, you can use the PTP Track Hound to analyse the PTP network traffic and verify it is working correctly.
I've also ordered a CM4 to add to my RPi collection. I hope I can do some hardware testing on my own as soon as that arrives :)
One option to get the Time Card working with IEEE1588-2008 before a driver is ready is to switch to software timestamping in the ptp4l config. Software timestamping is quite inaccurate though and doesn't do the accuracy of the Time Card justice.
Yeah I may go with that for now—the first and most important thing I wanted to test is whether the Pi would be able to use the driver. And it does! Second is to get it to do hardware timestamping—and it should be able to soon, once a driver supports TX too—but this is a secondary goal for now.
Like my quest to get an external GPU working on the Pi, I'm satisfied with small victories, and I know we're kind of riding on the bleeding edge of what's possible on a $25+ SBC ;)
I've also ordered a CM4 to add to my RPi collection. I hope I can do some hardware testing on my own as soon as that arrives :)
Exciting! I know you have a lot more experience than I do on PTP, and the more people can get testing (and eventually integrating these things in the wild), the better support will be, and the more interesting eventual use cases will emerge.
I know my Dad has used master clocks for audio sync in radio quite a bit... this kind of (slightly more) inexpensive open source solution would be right up his alley!
Posted a patch for the Debian compile issue in the Time Card repo: https://github.com/opencomputeproject/Time-Appliance-Project/pull/25
Edit: merged!
Do you have a GPS antenna connected? I see "utc_offset: 0 valid:0 leap_valid:0" in the kernel output above which implies it didn't have a GPS lock at bootup. The driver mentions one of the virtual serial ports is connected to the GPS output: "/sys/class/timecard/ocp0/ttyGNSS". Running gpsd+cgps on that serial port would let you see the signal strength and other status messages from the GPS.
Each PTP device has its own clock (named "PHC" in linux). You'll have a total of three different clocks here - the time card, the main system clock, and the BCM54210PE PHY's clock.
There's a few different ways of keeping those clocks in sync. I'm not seeing support for PPS input on the BCM54210PE PHY datasheet, so I assume you're going to have to sync time using phc2sys from linuxptp. I think ptp4l has a way of doing this through a config setting, but I'm don't know it off the top of my head.
You can test running a sync applying the main system clock (CLOCK_REALTIME) as the source of time (-s) and the time card (ptp0) as the destination (-c). You may also want to try the other direction if your time card has a GPS lock.
$ sudo phc2sys -E linreg -s CLOCK_REALTIME -c /dev/ptp0 -O 0 -m phc2sys[2394983.710]: /dev/ptp0 sys offset -319 s0 freq +26148 delay 803 phc2sys[2394984.711]: /dev/ptp0 sys offset -314 s0 freq +26148 delay 898 phc2sys[2394985.711]: /dev/ptp0 sys offset -397 s0 freq +26148 delay 800 phc2sys[2394986.711]: /dev/ptp0 sys offset -454 s2 freq +25655 delay 818 phc2sys[2394987.711]: /dev/ptp0 sys offset -21 s2 freq +26057 delay 800 phc2sys[2394988.711]: /dev/ptp0 sys offset 22 s2 freq +26111 delay 803 phc2sys[2394989.711]: /dev/ptp0 sys offset 50 s2 freq +26143 delay 803 phc2sys[2394990.712]: /dev/ptp0 sys offset 6 s2 freq +26126 delay 803
offset and delay are in nanoseconds, frequency is in parts per billion or nanoseconds per second.
New video posted today: https://www.youtube.com/watch?v=tU0xC1ynaT8
And to @ddrown - my first goal was getting the card functioning on the Pi; I haven't yet had a chance to set it up closer to a window (my office is as far as you can get from any part of the home's exterior!), so you're correct that I didn't have the antenna plugged in.
I'm going to do some testing with my Intel I340 in the meantime, to see if I can get PTP through it and have this card plugged in through a PCIe switch / riser.
@ahmadexp just posted PTPBox for Pi CM4 with Intel i350 and I'll be checking that out soon!
Back at it today... a lot has changed.
Like:
- https://github.com/raspberrypi/linux/commit/129b03a77347a09d07c143229fc2ddd27b0d7a36 (add PTP support for CM4)
- https://github.com/raspberrypi/linux/commit/a004834f716de41a111723b95208c40af61999b2 (add PPS out)
Flashing the latest Pi OS version (4-4-2022, 64-bit, lite), ethtool still shows:
pi@tap:~ $ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
software-transmit
software-receive
software-system-clock
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none
After running sudo apt update && sudo apt dist-upgrade -y today, it shows the same.
After running sudo rpi-update (had to do it in a kinda weird way), I get:
pi@tap:~ $ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
hardware-transmit
hardware-receive
hardware-raw-clock
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
off
on
onestep-sync
onestep-p2p
Hardware Receive Filter Modes:
none
ptpv2-event
Running:
sudo ./testptp -d /dev/ptp0 -L0,2
sudo ./testptp -d /dev/ptp0 -p 1000000000
Gives me a PPS output on pin 9:
Now I need to see if more of the Time Card hardware is happy with the latest driver version(s).
Hey! Can you share more details about how you got it running?
I tried to get it on the 1GB CM4 module. Recompiled the latest stable 5.15.56-v8+ and the 5.19 kernel (thanks to your AWESOME guide/docker), but the driver failed to load with the call trace:
[ 4.419550] Call trace:
[ 4.419556] pci_irq_vector+0x84/0xa8
[ 4.419569] ptp_ocp_register_ext+0xc0/0x110 [ptp_ocp]
[ 4.419602] ptp_ocp_probe+0x278/0x7a4 [ptp_ocp]
[ 4.419623] pci_device_probe+0xc0/0x190
[ 4.419638] really_probe+0xc0/0x318
[ 4.419653] __driver_probe_device+0x80/0xe8
[ 4.419664] driver_probe_device+0xb8/0x118
[ 4.419676] __driver_attach+0x78/0x110
[ 4.419687] bus_for_each_dev+0x7c/0xd0
[ 4.419697] driver_attach+0x2c/0x38
[ 4.419707] bus_add_driver+0x194/0x208
[ 4.419718] driver_register+0x6c/0x128
[ 4.419730] __pci_register_driver+0x4c/0x58
[ 4.419744] ptp_ocp_init+0x70/0x1000 [ptp_ocp]
[ 4.419764] do_one_initcall+0x54/0x2a0
[ 4.419778] do_init_module+0x50/0x228
[ 4.419793] load_module+0x2184/0x27e8
[ 4.419806] __do_sys_init_module+0x1a0/0x200
[ 4.419821] __arm64_sys_init_module+0x24/0x30
[ 4.419836] invoke_syscall+0x4c/0x110
[ 4.419850] el0_svc_common.constprop.3+0x98/0x120
[ 4.419864] do_el0_svc+0x2c/0x90
[ 4.419877] el0_svc+0x24/0x60
[ 4.419888] el0t_64_sync_handler+0x90/0xb8
[ 4.419899] el0t_64_sync+0x180/0x184
[ 4.419909] ---[ end trace a2f0621554062b60 ]---
[ 4.419926] ptp_ocp 0000:01:00.0: Could not get irq 1
[ 4.420277] ptp_ocp: probe of 0000:01:00.0 failed with error -22
Also, the Time Card seems to have issues with some PCIe expanders - when I tried the PCIe x1 to x16 "over USB cable" adapter, the Pi couldn't complete the PCIe enumeration when TC was plugged in.
Update: It partially works with the 2 GB CM4 module but doesn't at all with 1 GB. On CM4 it's limited (for some reason) to a single interrupt. Hence the driver only exposes the PTP device and no advanced features, like the GNSS TTY or NMEA TTY.
03:00.0 Memory controller: Facebook, Inc. Device 0400
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin ? routed to IRQ 39
Region 0: Memory at 600000000 (32-bit, non-prefetchable) [size=32M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/32 Maskable- 64bit+
Address: 00000000fffffffc Data: 6540
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 1, Latency L0s <64ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 26.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [100 v1] Device Serial Number 00-00-00-00-00-00-00-00
Kernel driver in use: ptp_ocp
Kernel modules: ptp_ocp
and the dmesg:
[ 4.278190] ptp_ocp 0000:03:00.0: enabling device (0000 -> 0002)
[ 4.279502] ptp_ocp 0000:03:00.0: irq 1 out of range, skipping ts0
[ 4.279531] ptp_ocp 0000:03:00.0: irq 2 out of range, skipping ts1
[ 4.279546] ptp_ocp 0000:03:00.0: irq 6 out of range, skipping ts2
[ 4.279559] ptp_ocp 0000:03:00.0: irq 15 out of range, skipping ts3
[ 4.279572] ptp_ocp 0000:03:00.0: irq 16 out of range, skipping ts4
[ 4.279693] ptp_ocp 0000:03:00.0: irq 11 out of range, skipping signal_out[0]
[ 4.279708] ptp_ocp 0000:03:00.0: irq 12 out of range, skipping signal_out[1]
[ 4.279721] ptp_ocp 0000:03:00.0: irq 13 out of range, skipping signal_out[2]
[ 4.279733] ptp_ocp 0000:03:00.0: irq 14 out of range, skipping signal_out[3]
[ 4.280026] ptp_ocp 0000:03:00.0: irq 7 out of range, skipping i2c_ctrl
[ 4.280040] ptp_ocp 0000:03:00.0: irq 3 out of range, skipping gnss_port
[ 4.280052] ptp_ocp 0000:03:00.0: irq 4 out of range, skipping gnss2_port
[ 4.280064] ptp_ocp 0000:03:00.0: irq 5 out of range, skipping mac_port
[ 4.280077] ptp_ocp 0000:03:00.0: irq 10 out of range, skipping nmea_port
[ 4.280089] ptp_ocp 0000:03:00.0: irq 9 out of range, skipping spi_flash
[ 4.359245] ptp_ocp 0000:03:00.0: Version 1.2.0, clock PPS, device ptp1
[ 4.359297] ptp_ocp 0000:03:00.0: Time: 1660295049.472829090, in-sync
Using
ptp4landphc2sysviasudo apt-get install -y linuxptp:pi@cm4:~ $ sudo systemctl start ptp4l pi@cm4:~ $ sudo systemctl status ptp4l ● ptp4l.service - Precision Time Protocol (PTP) service Loaded: loaded (/lib/systemd/system/ptp4l.service; disabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2021-08-20 15:59:59 CDT; 29s ago Docs: man:ptp4l Process: 692 ExecStart=/usr/sbin/ptp4l -f /etc/linuxptp/ptp4l.conf -i eth0 (code=exited, status=255/EXCEPTION) Main PID: 692 (code=exited, status=255/EXCEPTION) Aug 20 15:59:59 cm4 systemd[1]: Started Precision Time Protocol (PTP) service. Aug 20 15:59:59 cm4 ptp4l[692]: [238.440] interface 'eth0' does not support requested timestamping mode Aug 20 15:59:59 cm4 ptp4l[692]: failed to create a clock Aug 20 15:59:59 cm4 systemd[1]: ptp4l.service: Main process exited, code=exited, status=255/EXCEPTION Aug 20 15:59:59 cm4 systemd[1]: ptp4l.service: Failed with result 'exit-code'.So I'll get back to that later.
So I've been working to get PTP working on my recently obtained CM4 and Uputronics GPS HAT. Interestingly, when I installed linuxptp, it does not appear to install a ptp4l service instead a service for timemaster is installed. FWIW, I do already have chrony installed and running. chrony is configured to utilize hardware timestamps.
● timemaster.service - Synchronize system clock to NTP and PTP time sources
Loaded: loaded (/lib/systemd/system/timemaster.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2022-08-13 19:20:45 UTC; 7min ago
Docs: man:timemaster
Main PID: 1989 (timemaster)
Tasks: 2 (limit: 2197)
CPU: 89ms
CGroup: /system.slice/timemaster.service
├─1989 /usr/sbin/timemaster -f /etc/linuxptp/timemaster.conf
└─1990 /usr/sbin/chronyd -n -f /var/run/timemaster/chrony.conf
Aug 13 19:20:45 etu systemd[1]: Started Synchronize system clock to NTP and PTP time sources.
Aug 13 19:20:45 etu timemaster[1989]: [2474.652] process 1990 started: /usr/sbin/chronyd -n -f /var/run/timemaster/chrony.conf
Aug 13 19:20:45 etu chronyd[1990]: chronyd version 4.0 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEB>
Aug 13 19:20:45 etu chronyd[1990]: Enabled HW timestamping (TX only) on eth0
Aug 13 19:20:45 etu chronyd[1990]: Frequency 19.469 +/- 0.006 ppm read from /var/lib/chrony/chrony.drift
Aug 13 19:20:45 etu chronyd[1990]: Using right/UTC timezone to obtain leap second data
Aug 13 19:20:50 etu chronyd[1990]: Selected source 3.220.42.39 (virginia.time.system76.com)
Aug 13 19:20:50 etu chronyd[1990]: System clock TAI offset set to 37 seconds
Aug 13 19:21:32 etu chronyd[1990]: Selected source GPS
Start with sudo rpi-update as the ptp-enabled driver is not yet in the mainstream.
If that fails - paste the ethtool -T result.
@MaciekMachni
I've updated to the latest kernel. Linux etu 5.15.56-v8+ #1575 SMP PREEMPT Fri Jul 22 20:31:26 BST 2022 aarch64 GNU/Linux
ethtool -T eth0 returns:
Time stamping parameters for eth0:
Capabilities:
hardware-transmit
hardware-receive
hardware-raw-clock
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
off
on
onestep-sync
onestep-p2p
Hardware Receive Filter Modes:
none
ptpv2-event
Based on @geerlingguy blog post, this looks to be correct.
Side-note: I figured out that the installed /etc/linuxptp/timemaster.conf incorrectly refers to /etc/chrony.conf instead of /etc/chrony/chrony.conf
Next, it looks like I need to set the ptp_domain so something like this in timemaster.conf:
[ptp_domain 0]
interfaces eth0
delay 10e-6
as well as some modifications in chrony.conf:
## PTP
refclock PHC /dev/ptp0 tai
hwtimestamp *
hwclockfile /etc/adjtime
This results in
● timemaster.service - Synchronize system clock to NTP and PTP time sources
Loaded: loaded (/lib/systemd/system/timemaster.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2022-08-14 00:36:13 UTC; 12s ago
Docs: man:timemaster
Main PID: 1069 (timemaster)
Tasks: 4 (limit: 2197)
CPU: 134ms
CGroup: /system.slice/timemaster.service
├─1069 /usr/sbin/timemaster -f /etc/linuxptp/timemaster.conf
├─1070 /usr/sbin/chronyd -n -f /var/run/timemaster/chrony.conf
├─1071 /usr/sbin/ptp4l -l 5 -f /var/run/timemaster/ptp4l.0.conf -H -i eth0
└─1072 /usr/sbin/phc2sys -l 5 -a -r -R 1.00 -z /var/run/timemaster/ptp4l.0.socket -t [0:eth0] -n 0 -E ntpshm -M 0
Aug 14 00:36:13 etu chronyd[1070]: chronyd version 4.0 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +NTS +SECHASH +IPV6 -DEB>
Aug 14 00:36:13 etu ptp4l[1071]: [2577.819] [0:eth0] port 1: INITIALIZING to LISTENING on INIT_COMPLETE
Aug 14 00:36:13 etu ptp4l[1071]: [2577.819] [0:eth0] port 0: INITIALIZING to LISTENING on INIT_COMPLETE
Aug 14 00:36:13 etu chronyd[1070]: Enabled HW timestamping (TX only) on eth0
Aug 14 00:36:13 etu chronyd[1070]: Frequency 18.997 +/- 0.462 ppm read from /var/lib/chrony/chrony.drift
Aug 14 00:36:13 etu chronyd[1070]: Using right/UTC timezone to obtain leap second data
Aug 14 00:36:14 etu phc2sys[1072]: [2578.816] [0:eth0] Waiting for ptp4l...
Aug 14 00:36:18 etu chronyd[1070]: Selected source 52.10.183.132 (oregon.time.system76.com)
Aug 14 00:36:18 etu chronyd[1070]: System clock TAI offset set to 37 seconds
Aug 14 00:36:20 etu ptp4l[1071]: [2584.936] [0:eth0] selected local clock e45f01.fffe.d31bf2 as best master
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
#? NMEA 0 4 0 513 +1261us[+3353us] +/- 100ms
#* GPS 0 4 0 513 -1774ns[+2090us] +/- 200ns
#x PHC2 0 4 377 12 +37.0s[ +37.0s] +/- 112ns
#x PTP0 0 2 377 3 +71ms[ +71ms] +/- 930us
^- time.cloudflare.com 3 6 377 13 -1163us[-1163us] +/- 31ms
^- virginia.time.system76.c> 2 6 377 12 -908us[ -908us] +/- 37ms
^- ohio.time.system76.com 2 6 377 12 -2176us[-2176us] +/- 39ms
^- oregon.time.system76.com 2 6 377 13 +1432us[+1432us] +/- 30ms
but sudo phc2sys -E linreg -s CLOCK_REALTIME -c /dev/ptp0 -O 0 -m shows the offsets being all over and not the recommended < 100
phc2sys[2677.212]: /dev/ptp0 sys offset -1473165 s0 freq +704 delay 57462
phc2sys[2678.216]: /dev/ptp0 sys offset -1474345 s0 freq +704 delay 58924
phc2sys[2679.221]: /dev/ptp0 sys offset -1473787 s0 freq +704 delay 66296
phc2sys[2680.227]: /dev/ptp0 sys offset -1473601 s1 freq +393 delay 66999
phc2sys[2681.233]: /dev/ptp0 sys offset -1492922 s2 freq -1491350 delay 65925
phc2sys[2682.238]: /dev/ptp0 sys offset 4725 s2 freq -738804 delay 66258
phc2sys[2683.243]: /dev/ptp0 sys offset 748670 s2 freq +4042 delay 53091
phc2sys[2684.247]: /dev/ptp0 sys offset 746168 s2 freq +746275 delay 52832
phc2sys[2685.251]: /dev/ptp0 sys offset -426 s2 freq +1061 delay 54017
phc2sys[2686.256]: /dev/ptp0 sys offset -3174 s2 freq -1258 delay 61629
phc2sys[2687.261]: /dev/ptp0 sys offset -1374 s2 freq -1072 delay 62832
phc2sys[2688.265]: /dev/ptp0 sys offset 484 s2 freq -89 delay 55277
phc2sys[2689.270]: /dev/ptp0 sys offset 1069 s2 freq +1727 delay 62610
phc2sys[2690.274]: /dev/ptp0 sys offset -169 s2 freq +468 delay 53147
phc2sys[2691.278]: /dev/ptp0 sys offset -100 s2 freq +392 delay 53277
phc2sys[2692.282]: /dev/ptp0 sys offset 98 s2 freq +612 delay 53184
phc2sys[2693.287]: /dev/ptp0 sys offset 100 s2 freq +663 delay 61555
phc2sys[2694.292]: /dev/ptp0 sys offset 102 s2 freq +600 delay 61628
phc2sys[2695.296]: /dev/ptp0 sys offset -125 s2 freq +483 delay 52962
phc2sys[2696.300]: /dev/ptp0 sys offset 51 s2 freq +605 delay 52870
phc2sys[2697.304]: /dev/ptp0 sys offset -218 s2 freq +476 delay 53184
phc2sys[2698.308]: /dev/ptp0 sys offset -51 s2 freq +516 delay 52962
Compiling https://tsn.readthedocs.io/_downloads/f329e8dec804247b1dbb5835bd949e6f/check_clocks.c returns
Dumping timestamps and deltas
rt tstamp: 1660437706379766701
tai tstamp: 1660437743379766905
phc tstamp: 1660437706379901358
rt latency: 93
tai latency: 111
phc latency: 697399
phc-rt delta: 134657
phc-tai delta: -36999865547
phc-rt delta is not 37 sec !
phc-tai delta is greater than 50 usec !
Failed to send PTP management packet - No such file or directory
Failed to send PTP management packet - No such file or directory
Please verify ptp4l and phc config and restart them if necessary to synchronize the clocks !
In order to use chrony's hardware timestamps for NTP packets, you'll need to use the experimental NTP in PTP mode.
This output from ethtool shows the problem:
Hardware Receive Filter Modes:
none
ptpv2-event
The CM4's PHY only has hardware timestamps for ptp packets.
The relevant config in chrony.conf:
hwtimestamp eth0 rxfilter ptp
ptpport 319
and to send ntp-in-ptp packets to a given server, add "port 319"
server ntp.local minpoll 0 maxpoll 4 xleave port 319
If you have the measurements log enabled, you can see what type of timestamps were used for each sample. K=kernel, H=hardware
Thanks @ddrown. Unfortunately, the as-shipped version of chrony in the current Raspberry Pi OS appears to be 4.0 and the feature you are referring to appears to be post-4.0. I'll have to look to see if there is a newer build somewhere.