raspberrypi
CM4 is missing IEEE1588-2008 support through BCM54210PE
Is this the right place for my bug report? Yes: I asked on the forum, I asked the raspberry foundation through the contact form but didn't get any response - it is very much kernel related as is explained in the forum post.
Describe the bug The CM4 has an undocumented inofficial PHY (BCM54210PE) that supports hardware timestamping (and a hardware clock, actually??), but there seems to be no driver supporting these features.
So CM4 does actually not provide hardware based IEEE1588-2008 support as opposed to what the raspberry pi foundation actually communicates in the CM4 datasheet.
To reproduce
sudo ethtool -T eth0 only shows software timestamping capabilities.
Expected behaviour Depends - the raspberry pi foundation never communicated what IEEE1588-2008 features are supported such that is is unclear what exactly can be expected; also the PHY documentation is not available such that possibilities (ie optimal expected behaviour) can not be described.
Expected (according to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-using_ptp):
- eth support for:
- SOF_TIMESTAMPING_TX_HARDWARE
- SOF_TIMESTAMPING_RX_HARDWARE
- (possibly) SOF_TIMESTAMPING_RAW_HARDWARE
- shows PTP Hardware Clock (because BCM54210PE has on-chip clock)
- Hardware Transmit Timestamp Modes ??
- Hardware Receive Filter Modes ??
The CM4 datasheet lists SYNC_IN, SYNC_OUT pins ment for IEEE1588-2008 support but has not further specified this. It actually is a question wether these pins are needed in the first place - because the PHY would seem to have a hardware-clock itself. Further it must be clear wether the PHY's internal clock (if any) is routed to a CM4 pin in any form and if not this should be fixed in future revisions because custom boards for the CM4 might require tight synchronization to a hardware clock.
Actual behaviour
sudo ethtool -T eth0 only shows software timestamping capabilities.
There is no (documented) synchronizable IEEE1588-2008 clock signal going from the CM4.
System Raspberry Pi Compute Module 4
Raspberry Pi OS Lite, January 11th 2021, Kernel version: 5.4 Ubuntu Server 20.04.2 LTS,
Logs none
Additional context BCM54210PE documentation is required to implement to see actual features provided and configuration options - IEEE802.3-2018 seems to define said capabilities etc. Possibly also required are CM4 schematics and BCM2711 documentation. Also see IEEE802.3-2018 Section 90. Ethernet support for time synchronization protocols
I can try to take care of the implementation, but I would like to have the mentioned documents.
Just compiling a kernel with PTP timestamping support enabled (by default not). Will see if this changes anything.
I'm new to kernel topics, so I missed the obvious. Will report back
Built a new kernel according to docs and changed the following settings (through make menuconfig):
CONFIG_NETWORK_PHY_TIMESTAMPING=y
#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=m
# CONFIG_DP83640_PHY is not set
# CONFIG_PTP_1588_CLOCK_INES is not set
# CONFIG_PTP_1588_CLOCK_IDT82P33 is not set
# CONFIG_PTP_1588_CLOCK_IDTCM is not set
# end of PTP clock support
Linux cm4-1 5.10.14-v7l-MY_CUSTOM_KERNEL+ #1 SMP Tue Feb 16 14:05:19 GMT 2021 armv7l GNU/Linux
No change observed, ie
pi@cm4-1:~/linux $ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
software-transmit (SOF_TIMESTAMPING_TX_SOFTWARE)
software-receive (SOF_TIMESTAMPING_RX_SOFTWARE)
software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none
pi@cm4-1:~ $ sudo lshw -class network
*-network:0
description: Ethernet interface
physical id: 2
logical name: eth0
serial: dc:a6:32:XX:XX:XX
size: 1Gbit/s
capacity: 1Gbit/s
capabilities: ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=bcmgenet driverversion=5.10.14-v7l-MY_CUSTOM_KERNEL+ duplex=full ip=192.168.2.197 link=yes multicast=yes port=MII speed=1Gbit/s
...
I figure because there is no particular PTP hardware clock (driver) there's no recognized PTP clock (...) - but this would depend on wether BCM54210PE actually would have a hardware clock (?)
bcmgenet does not seem to use phy_devices struct mii_timestamper *mii_ts;
so enabling timestamping has no effect...
For reference: https://github.com/raspberrypi/linux/blob/c9226080e513181ffb3909a905e9c23b8a6e8f62/include/linux/mii_timestamper.h
We're currently discussing getting support for writing a driver from Broadcom, although we have access to the datasheet, to me it's pretty meaningless!
Unfortunately, so is the Linux interface to the clock...
But we'll endeavour to get there
Ok, thanks!
Hoping to be hearing more from you!
Just to say, as for the docs, I wouldn't mind signing an NDA to have a look to see if can make sense of it..
Damn you Broadcom and your closed datasheets! Timestamping at the PHY level is the best one can get, but not if unimplemented in software...
I also do hope those SYNC_IN and SYNC_OUT pins are actual event inputs/outputs for external sync and were preserved on the I/O board and not repurposed for something else. I've seen countless SBCs where this potential was wasted.
So, @ghollingworth, aside from a PTP driver eventually being developed - do we know if the two pins mentioned (pins 16 and 18 on the CM4) are presented on the I/O board?
Thanks!
@wowczarek
From what I see in the CM4IO datasheet (page 7) SYNC_IN, SYNC_OUT are available on J2 pins 8+9
I saw @wowczarek message on TimeNuts. I used to work for Broadcom's Ethernet PHY group and worked on the 5421x series including the IEEE1588 part of the chip. I remain on good relations with my former colleagues. There are several packages and two dies which map to the BCM54210 part number. I think the part on the Raspberry Pi (BCM54213PE? is that correct) does not pin out the SYNC_IN and SYNC_OUT pins even though they are on the die (36 pin package). I will check with them. Even if the HW SYNC pins are not brought out, some functionality of the IEEE-1588 portion of the PHY may be able to be enabled.
@ghollingworth, does the datasheet you have access to mention IEEE-1588?
On the CM4 we use the BCM54210PE and the SYNC_IN and SYNC_OUT are exposed to pins on the CM4 connector. The CM4IO board exposes these signals on connector J2. @jaycordaro if the PHY team are looking at a driver for this it would be great.
ok I spoke to some friends who are still at Broadcom. The PHY does support HW timestamping, the register writes required to enable HW timestamping and the SYNC pins are described in an applications note, "IEEE-1588 for Timing Applications," not in the datasheet, so that may explain why the datasheet didn't provide anything useful. Someone who has Broadcom DocSafe access for BCM54210PE could request that document to be added to their account. The Product Line Manager would then approve the request. The PHY team people that I talked to (senior ASIC mgr for the PHY and someone in Marketing) are not aware of any effort inside the company to support a driver but they will try to help.
Thanks to everyone on this... Especially to Philphil on the forums and @jaycordaro for their help! I do have access to the register information and Broadcom Docsafe and am currently trying to play catchup with the PTP kernel interface and ioctl interface to understand what it is I need to implement to get a working timestamping system.
What's clear is that there are a number of interfaces implementing different functionality, just need to decide on the order of implementation.
What is it that people actually want out of this?
- external signal timestamping (timestamping of the SYNC_IN signal)
- external event creation (SYNC_OUT periodic output signal)
- TX and RX packet hardware timestamping (the BRCM MAC contains a FIFO from which you can read hardware timestamps)
- PTP Hardware clock (which I think is implied with 1 and 2 but maybe not 3
Gordon
@ghollingworth actually 4. is implied exactly in 3., at least from the kernel interface point of view, for network sync (IEEE 1588) to work.
What is needed is all of the items you list, but as to the order:
a. 4. is required because this is the interface to controlling the clock. b. 3. is required to sync this clock over the network c. 1. is required to sync with an external hardware pulse such as from GNSS receivers or other quality clocks d. 2. is required for external measurement and characterisation, which is priceless
- Items 1. and 2. are rare opportunities for hardware this common, giving this platform a chance to be the prime off-the-shelf testing ground for time sync related projects (*)
- Items 3. and 4. are the part that any other 1588-compliant chip has driver support for.
So unselfishly I'd say 3,4, then 1,2, but really it's the whole that people / we are / should be after.
(*) While we are at it, a set of u.fl (or ideally SMA) connectors could be considered for these two pins in the next revision of the carrier board instead, and freeing up the pins for something else. That would mean plug and play for serious timing, but it's not a big issue otherwise, we have the header. This is debatable though - a GNSS receiver on a board will also have a pin, but a more serious GNSS driven oscillator will have a coax connector.
Thanks!
Thanks that helps a lot.
If you've got a suggestion for an Ethernet device that does actually implement these functions from the standard set of drivers:
I would suggest one of these (i.e. drivers that reference the ptp_clock_register function)
https://elixir.bootlin.com/linux/latest/C/ident/ptp_clock_register
That would help me with an initial implementation...
Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...
many thanks also!
Well, simply put I would think that all features might serve one use case or other and thus - optimally - all should be available to make the CM4 an attractive platform for any of those use cases.
The PTP hardware clock would seem the most reliable/precise possible and thus if it would be available that would be a definite yes.
Sync'ing external components to the PTP clock is - as I would think - also a core feature 1588 capable platforms should offer; an example being where audio ADCs and DACs should be driven by the sampling frequency derived from the network clock (thinking of AES67 here). As I understand from your wording, the periodic signal is adjustable within a particular range/frequency?
Analogously sync'ing the PTP clock to an external component would be quite interesting a feature to have, but as I understand the way SYNC_IN works is just timestamp generation.
Knowing the precise timestamp of an event on an external component would seem as important; ex knowing the precise time of sample acquisition, in particular if external components are not sync'ed to the PTP clock.
So I would consider 1, 2 and 4 to be quite important features to have for a module aiming at deeply embedded applications and would make the CM4 an interesting base indeed.
As for 3 (selective TX/RX hardware timestamping) nothing immediately comes to mind (not working with such tech) but I'm sure there are meaningful use cases.
wowczarek: I'm not sure 3 is required for actual network/ptp syncing, depends a bit how the 1588 functionalities are implemented (on-chip?)
but yes, I agree, 1+2 are indeed rare opportunities for hardware this common and it's indeed the whole package that should be aimed for
@tschiemer indeed, so this is why I made a distinction of sorts in the items from Gordon's list. However, IEEE 1588 == network time sync, don't forget. Without hardware packet timestamping (3,4) you cannot have a working PTP stack.
Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...
Spontaneous ideas... use a (HW or peripheral) clock for comparison? Observe SYNC_OUT on some external component use SYNC_IN to generate a timestamp and compare values?.... if it's an on-PHY clock, you just have to see how it runs, I guess?
Ja, I'm aware, just don't know what magic the PHY is capable of by itself.. thinking bout it, it likely won't have a PTP stack of it's own.. :/
@ghollingworth
If you've got a suggestion for an Ethernet device that does actually implement these functions from the standard set of drivers:
I think the closest are Intel i2xx/i3xx etc. where there are the two SDIO pins that can be ordered to be event inputs / outputs. The Linux PTP/PHC API has a way to grab the input event timestamps I would prefer a /dev/ppsX device, but that's just me.
Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...
Yes, ptp4l for the h/w timestamping and PTP Hardware Clock. I mean really what the "PTP hardware clock" is, "merely" two things, plus some initialisation logic:
- A counter + register that counts either arbitrary ticks, with maybe another register to hold time of day or a lower/higher register pair for precise and rough time - or like Intel did it, a register/counter that ticks in human time. I am expecting the former. Plus some code to handle rollovers, etc.
- Another register that controls the counter increment - this is how you steer the clock.
The timestamps captured are obviously latched counter values with a FIFO. So the timestamping driver part will pick these up, associate with frames/packets and attach these as out of band mesages (SCM) to the socket kernel buffer.
So the PHC driver provides an interface to this, allowing us to read time, set time, and steer the clock. If you use ptp4l as a PTP slave ('slave' term is under debate these days), you will utilise h/w timestamps to compute offset and control the hardware clock, this is why timestamps and PHC are one item.
@tschiemer
Ja, I'm aware, just don't know what magic the PHY is capable of by itself.. thinking bout it, it likely won't have a PTP stack of it's own.. :/
It won't. The PHY only timestamps packets, that is it, you need a software stack like ptp4l.
-
The hardware clock interface or PTP Hardware Clock / PHC, in Linux kernel terms, is only a standard interface for reading, writing and steering increment of the time register(s) on the chip and converting to human time. In hardware terms it's a register or a few, and that is that. This is not the same as the in/out pins; these are an extra.
-
"IEEE 1588 support" means some time register as above, and hardware timestamping, which means only one thing: timestamping packets/frames. Either the ability to timestamp every packet, which Intel, Solarflare and some others support, or, more commonly, only timestamp PTP packets, so the hardware matches a PTP packet using either the multicast group destination MAC address or PTP Ethernet multicast address or link-local variants of those (right at the beginning of frame), or if it's more sophisticated, UDP port 319, which also allows for unicast IP PTP timestamping. Hardware matches these, latches the timestamp counter and pushes that up a FIFO, long story short, they end up being human time attached to packets arriving at the socket. Obviously I don't know this chip and what's in the magic document, so I don't know which of the above matching options it does. There are also things like VLAN tag matching. What I do know is Broadcom's switching ASICs, and their PTP support is solid and extensive.
-
The event input pin should generate a timestamp aligned with the PTP clock on input rise transition (well, typically rise, for most timing applications). I hope it will be that simple. TI's AM335x network subsystem for example does it differently - event input has its own counter, but luckily it can be correlated with the PTP counter on rollover - more work, but it works. The problem with the event input and testing it, is that because it's meant to be handled in the PTP API, it's mostly ignored by people with GNSS receivers and there is near-zero software support for it. Really it should be a /dev/ppsX device.
-
The event output pin should allow, at minimum and by default, a rising pulse of certain width at the top of every second, of a certain width - make it 100 ms to keep all sync applications happy.
Testing 3. and 4, at least preliminary, can be done with two boards back to back, but ideally you need a time interval counter / frequency counter. Time nuts around the world will handle the rest. Two boards, Ethernet back to back, crossed over event pins, one is a PTP master, the other a PTP slave (ptp4l), PTP syncs them, and the inputs will show the clock difference between the two. That's the minimal DUT/SUT setup.
Unfortunately, I was able to find out in December. I put an i210 in the PCIe slot and built a kernel. The PHY clock is synchronized and so is the board. Unfortunately the clock of the CM4 is not good enough and the whole thing runs away after hours and the clock can no longer be synchronized with the PHY .
@marcohabeck this is off-topic - but if you mean it runs away after being synchronised once and then leaving it, then welcome to the real world. In a PC or server it would be no different. This hardware can timestamp, but it can't keep time, that's just the way it is. Poor oscillators. Timing cards with oscillators good enough to hold better than hundreds of microseconds per day and few and far between. Oregano/Meinberg and high-end Exablaze are the only ones I know can do decent holdover. Solarflare right behind them. These systems have to be kept actively synchronised all the time, they drift very quickly.
But technically PTP NICs can be steered much more aggressively than the OS clock (I assume this is what you mean by 'clock' vs. PHY), so the CM4 or drivers or software would have had to be severely broken if the OS clock couldn't keep up with the NIC. OS clock control is limited to +/- 512ppm (~half a millisecond per second) adjustment, which is huge. People would have noticed drift of this magnitude. You'd have to check your software stack. Was the PHY (MAC really) synchronised to PTP, or were you just synchronizing the OS clock to the NIC without PTP?
I'm quite happy to help out with hardware-side testing on this stuff; the PHY hardware timestamping support is most of the reason I picked up a CM4+CM4IO board, and it would be absolutely killer for something like a next-gen LimeNET Micro (and for time sync applications in general)
I've got a number of PTP-equipped devices and 1PPS sources - some commercial, some homemade - available for testing, though I doubt you'd have much trouble finding plenty of people who're interested!
Hi @ghollingworth, I've been working on the ptp 1588 driver for the bcm54210pe, at the moment I've got the full skeleton implemented and the kernel can check configuration available for the PTP and PHC:
root@rpi-cm4:~# ethtool -T eth0
Time sta[ 64.191025] DEBUG: Passed bcm54210pe_ts_info 258
mping parameters for eth0:
Capabilities:
hardware-transmit (SOF_TIMESTAMPING_TX_HARDWARE)
hardware-receive (SOF_TIMESTAMPING_RX_HARDWARE)
hardware-raw-clock (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
off (HWTSTAMP_TX_OFF)
on (HWTSTAMP_TX_ON)
one-step-sync (HWTSTAMP_TX_ONESTEP_SYNC)
Hardware Receive Filter Modes: none
Also I can get and set the PHC using phc_ctl tool:
root@rpi-cm4:~# date
Fri Mar 9 13:03:13 UTC 2018
root@rpi-cm4:~# phc_ctl /dev/ptp0 get
[ 174.782856] DEBUG: Passed ptp_clock_gettime 109
[ 174.787636] DEBUG: Passed ptp_clock_gettime 114
[ 174.792269] DEBUG: bcm54210pe_gettime 176
[ 174.796722] DEBUG: Get time 4386 13124 21862 0 0
[ 174.801471] DEBUG: Current PCH time 860116326.0
[ 174.806145] DEBUG: Passed ptp_clock_adjtime 126
[ 174.810797] DEBUG: Passed ptp_clock_gettime 109
[ 174.815454] DEBUG: Passed ptp_clock_gettime 114
[ 174.820073] DEBUG: bcm54210pe_gettime 176
[ 174.824450] DEBUG: Get time 4386 13124 21862 0 0
[ 174.829163] DEBUG: Current PCH time 860116326.0
phc_ctl[174.832]: clock time is 1234549248713053542.000000000 or (null)
root@rpi-cm4:~# phc_ctl /dev/ptp0 set
[ 176.947297] DEBUG: Passed ptp_clock_gettime 109
[ 176.951944] DEBUG: Passed ptp_clock_gettime 114
[ 176.956614] DEBUG: bcm54210pe_gettime 176
[ 176.961005] DEBUG: Get time 4386 13124 21862 0 0
[ 176.965790] DEBUG: Current PCH time 860116326.0
[ 176.970450] DEBUG: Passed ptp_clock_adjtime 126
[ 176.975122] DEBUG: Passed ptp_clock_settime 98
[ 176.979653] DEBUG: Original time 1520600486 102643396
[ 176.984846] DEBUG: Set time 0 23202 34214 1552 14020
phc_ctl[176.988]: set clock time to 1520600486.102643396 or Fri Mar 9 13:01:26 2018
root@rpi-cm4:~# phc_ctl /dev/ptp0 get
[ 179.635848] DEBUG: Passed ptp_clock_gettime 109
[ 179.640502] DEBUG: Passed ptp_clock_gettime 114
[ 179.645127] DEBUG: bcm54210pe_gettime 176
[ 179.649521] DEBUG: Get time 0 23202 34214 1552 14020
[ 179.654601] DEBUG: Current PCH time 1520600486.101725892
[ 179.660022] DEBUG: Passed ptp_clock_adjtime 126
[ 179.664650] DEBUG: Passed ptp_clock_gettime 109
[ 179.669267] DEBUG: Passed ptp_clock_gettime 114
[ 179.673883] DEBUG: bcm54210pe_gettime 176
[ 179.678254] DEBUG: Get time 0 23202 34214 1552 14020
[ 179.683331] DEBUG: Current PCH time 1520600486.101725892
phc_ctl[179.687]: clock time is 1520600486.101725892 or Fri Mar 9 13:01:26 2018
The current blocking point is that, despite reading and setting the PHC using phc_ctl set/get tool, this counter is not increasing its value, so it's not counting. Has any of the contributors in here got a contact in Broadcom in order to know whats happening with the register configuration? I've checked several times and I think it's correct. I cannot get any more ideas from the datasheet.
In parallel, I'm currently developing the time stamping routines in order to get the info from the Broadcom internal FIFO. I think I will have the first draft maybe during next week.
I'll be glad to contribute to this issue. Maybe we can create an branch in order to start collaborating here.
I have access to support from Broadcom and have been using it to get the PHY setup correctly, currently I've got it set up timestamping packets and working correctly, but am also having trouble reading back the current timer counter. Whick register set are you using? The original time stamp or the heartbeat time stamp registers?
Gordon
I'm currently using the Original Time register, I can read and write in it, but it's not counting.
Mi register setup is the following one (extracted from Broadcom .vbs scripts and the other is added from my side in order to configure the SYNC_OUT)
REGISTER, VALUE
0x0F5, 0x0001
0xA10, 0x0101
0xAF8, 0x0101
0xAC0, 0x0F0F
0xAC1, 0x0F0F
0xAC3, 0x0101
0xA21, 0x0001
0xA22, 0x0000
0xA54, 0x0000
0xA55, 0x0000
0xA56, 0x5566
0xA57, 0x3344
0xA58, 0x1122
0xA5C, 0x07C0
0xA5D, 0x07C0
0x82D, 0x407E
0xA5E, 0x0002
0xA5B, 0x0160
0xA11, 0x00FF
0xA19, 0x0000
0xA78, 0x8000
0xA79, 0x400
0xA7A, 0x240
0xA7F, 0xD012 => In order to work with internal sync and SYNC_OUT pulses. We've got nothing connected on SYNC_IN pin.
Please @ghollingworth tell me how can I send you my code as soon as I've got green light to share it. Thanks, Carlos