That's what the chrony tempcomp directive is for. But you would have to figure out the coefficients, it's not automatic.
An advantage of constantly loading at least one core of the CPU might be preventing the deeper power states from kicking in, which should make the RX timestamping latency more stable and improve stability of synchronization of NTP clients.
The major Linux distributions replaced ntp with ntpsec. A better question would be who is still running ntp. I know about FreeBSD and NetBSD.
ntpsec as a project seems to be doing ok. They are releasing new versions, fix reported issues, accept patches, and develop the code publicly. While ntp still has a huge list of acknowledged but unfixed CVEs.
Current Debian, Ubuntu, Fedora, RHEL/CentOS (EPEL) have an ntpsec package, but no ntp package. It's not used by default (that's chrony on most of the distributions), but the users can install it and use it.
Steering the same PHC with phc2sys as chronyd is using for HW timestamping is not the best approach as that creates a feedback loop (instability). It would be better to leave the PHC running free and just compare the sys<->PHC with PHC<->PPS offsets.
> So your 100m->1G link for example will already introduce over 1 us of error (to accuracy!), but NTP will never show you this
That doesn't apply to NTP to such an extent as PTP because it timestamps end of the reception (HW RX timestamps are transposed to the end of the packet), so the asymmetries in transmission lengths due to different link speeds should cancel out (unless the switches are cutting through in the faster->slower link direction, but that seems to be rare).
Yes, I saw those PTP posts and where the methodology lacks a bit.
Re. asymmetries canceling out, OK, I oversimplified, and this is true in theory and often in practice, but for example, having done this with nearly all generations of enterprise type Broadcom ASICs sort of 2008 onwards I know that there are so many variations to this behaviour that the only way to know is to precisely measure latencies in one direction and the other for a variety of speed, CT vs. S&F and frame sizes, and even bandwidth combinations and see. I used to characterise switches for this, build test harnesses, measurement tools etc., and I saw everything ranging from: CT one way, S&F the other way, but not for all speed combinations, then CT behaviour regardless of enabling or disabling it, finally even things like latency having quantised step characteristics in increments of say X bytes because internally the switching fabric used X-byte cells, and then CT only behaving like CT above certain frame sizes. There's just a lot to take into account. There are even cases where a certain level of background traffic _improves_ latency fairness and symmetry, an equivalent of keeping the caches hot.
The author's best bet at reliable numbers would be to get himself a Latte Panda Mu or another platform with TGPIO and measure against 1PPS straight from the CPU. That would be the ultimate answer. Failing that, at least a PTM NIC synced to the OS clock, but that will alter the noise profile of the OS clock.
But you and me know all this because we've been digging deep into the hardware and software guts of those things for years, and have done this for a job, and what's a home lab user to do. It's a never-ending learning exercise and the key is to acknowledge the possible unknowns, and by that I don't mean scientific unknowns but that we don't know what we don't know, and bloggers sometimes don't do this.
> Steering the same PHC with phc2sys as chronyd is using for HW timestamping is not the best approach as that creates a feedback loop (instability).
This is standard practice, though, for most PTP slave clocks. The feedback is just factored into the math. (Why? No idea. I just know how the code works.)
Although… it's standard practice in PTP setups that are designed for it. Not NTP… if only there were a specification… :)
I do have to wonder though. Of what use are timestamps from an unsynchronized PHC to chrony? Is it continuously taking twin sys+PHC timestamps to line up things?
> I do have to wonder though. Of what use are timestamps from an unsynchronized PHC to chrony? Is it continuously taking twin sys+PHC timestamps to line up things?
That would be the logical way to do it. You want the lowest jitter timestamps you can get on the incoming ethernet frames. If conditions are stable enough you can compute a timebase translation between your MAC local clock and sys clock using the best available method, potentially using many samples over a (relatively) long time frame. And as the GP says, this gives a feed-forward structure without any need for stabilising a feedback loop.
gPTP full-duplex ethernet peer synchronisation uses timestamps from free-running local clocks at each end of the link.
> gPTP full-duplex ethernet peer synchronisation uses timestamps from free-running local clocks at each end of the link.
Do you mean free-running as opposed to VCXO? Most implementations I know indeed use free-running clocks, but there's an increment/rate register in hardware that specifies what value to add to the time counter per crystal tick, which gets updated by the PTP layer — and timestamps use that. So even though the crystal is physically free running, the feedback loop is still there, it just doesn't include the crystal itself.
I know that ethernet MACs typically provide a dynamic mechanism to change the fractional phase increment of the timestamp counter, but it is clear from the gPTP specification that what is referred to as the "Local Clock" at each station is free running with respect to the PTP time base and with respect to connected peers. "Local Clock" is the time used to timestamp packet arrival/transmit times. "PTP Time" (i.e. TAI if we're GPS synchronised) is an independent logical time, separate from Local Time at each station.
It is very clear that there is no feedback loop in 802.1AS (gPTP). I can't speak to other PTP versions. Local clocks (used, among other things for peer delay estimation) are specified to be free running with respect to the PTP time base and to connected peers, not disciplined to PTP Time, asynchronous, not synchronised, not syntonised. The peer delay mechanism equations compensate for both time offset and rate difference in peer local clocks. Furthermore, in order to average estimates over multiple measurements, time offset and rate difference are assumed to be stable (i.e. messing with the rate of a local clock violates invariant assumptions of the peer delay algorithm).
A few more qualifications: I am talking here about non-master peers. I think it might be compliant for the grand master to discipline the local clock to the time source (e.g. GPS PPS), provided it appears stable to connected peers, but it is not at all required by the protocol. Similarly, you might discipline Local Clock to some other stable clock, or to perform temperature compensation. In principle you could synchronise Local Clock to system clock (both free-running with respect to PTP time) so that your packet timestamps are automatically in sys-clock timebase. Once again, there is nothing in the PTP spec that requires this, but the potential utility is clear on a general purpose OS (not necessarily so on an embedded device).
There's only one clock in HW, not two. And you really want PTP time in HW for PPS/timestamp IO. (And gPTP uses 1588 HW, there are no special 802.1AS HW implementations that I'm aware of.)
Whether this matches the spec — no idea. My knowledge is from implementations… there could of course be ones that have two clocks. Can you link one?
I have seen enough confusion in hardware specs that I would only trust the IEEE spec.
In gPTP there is one hardware clock, yes. The Local Clock. PTP time is not a hardware clock, it is a virtual clock. The Local Clock is not synchronised to PTP time. Do you have a copy of 802.1AS-2020 there? Here are a few quick quotes resulting from searching for "local clock":
"3.16 local clock: A free-running clock, embedded in a respective entity (e.g., PTP Instance, CSN node), that provides a common time to that entity relative to an arbitrary epoch." (p. 21)
"Each PTP Instance measures, at each PTP Port, the ratio of the frequency of the PTP Instance at the other end of the link attached to that PTP Port to the frequency of its own clock. The cumulative ratio of the Grandmaster Clock frequency to the local clock frequency is accumulated in a standard organizational type, length, value (TLV) attached to the Follow_Up message (or the Sync message if the optional one-step
processing is enabled). The frequency ratio of the Grandmaster Clock relative to the local clock is used in computing synchronized time, and the frequency ratio of the neighbor relative to the local clock is used in correcting the propagation time measurement." (p. 44)
"10.1.2.1 LocalClock entity
The LocalClock entity is a free-running local clock (see 3.16) that provides a common time to the PTP Instance, relative to an arbitrary epoch. A PTP Instance contains a LocalClock entity. The requirements for the LocalClock entity are specified in B.1. All timestamps are taken relative to the LocalClock entity (see 8.4.3). The LocalClock entity also provides the value of currentTime (see 10.2.4.12), which is used in the
state machines to specify the various timers.
NOTE—The epoch for the LocalClock entity can be the time that the PTP Instance is powered on.
" (p.66)
Need I continue?
I may be mistaken, but I thought in this sub-thread we were talking about the PTP clock and the system clock (e.g. the x86 tsc). Only one of these is relevant to PTP but the system clock is the only one available to timestamp software events and so you may want to be able to convert between tsc time, NIC local clock time, or PTP time. If you want to schedule GPIO events on the NIC, given a PTP time you can compute the corresponding local clock time, and schedule the event on the GPIO. This does not require disciplining the NIC clock (i.e. the PTP local clock, as defined in my quotes above) to PTP time.
chrony can be configured to encapsulate NTP messages in PTP messages (NTP over PTP) in order to get the delay corrections from switches working as one-step PTP transparent clocks. The current NTPv5 draft specifies an NTP-specific correction field, which switches could support in future if there was a demand for it.
The switches could also implement a proper HW-timestamping NTP server and client to provide an equivalent to a PTP boundary clock.
PTP was based on a broadcast/multicast model to reduce the message rate in order to simplify and reduce the cost of HW support. But that is no longer a concern with modern HW that can timestamp packets at very high rates, so the simpler unicast protocols like NTP and client-server PTP (CSPTP) currently developed by IEEE might be preferable to classic PTP for better security and other advantages.
That's way easier to fix, though. You just need to fix the kernel's internal structures to keep track of time using two numbers. Y2K was a problem because everyone had to fix their code, not just OSs.
It'll still be a problem for code that relies on time being 64-bit int of nanoseconds. How do you change the internal representation here without also breaking the API contract for application code?
My current understanding of the wireless technologies wrt to home automation:
- wifi is most reliable, secure, easiest to debug, but usable only for mains-powered devices due to higher energy consumption
- bluetooth LE has lowest energy consumption, best for unreliable broadcasting of data (e.g. temperature sensors), but has shorter range
- zigbee is best for battery-powered devices where reliable communication is needed and is initiated by the device (e.g. switch, window/door sensor)
- zwave is best for battery-powered devices which need to quickly receive data (e.g. door lock or directly controller radiator valves), but security seems problematic according to some reports
Regardless of theory, my experience has been that WiFi communications are significantly less reliable than Zigbee or BLE, and typically have worse latency to boot.
Unless you have a bunch of WiFi APs around your house, practical range with Zigbee is also better than WiFi because powered Zigbee nodes (like lightbulbs and switched outlets) act as mesh routers.
One additional benefit of Zigbee and Wave is the possibility of creating battery-less devices, like the Philips Hue Tap switch, which uses a hammer and piezoelectric generator for power.
To me that looks like they are reinventing NTP, but not addressing all the issues of PTP.
A big problem with the PTP unicast mode is an almost infinite traffic amplification (useful for DDoS attacks). The server is basically a programmable packet generator. Never expose unicast PTP to internet. In SPTP that seems to be no longer the case (the server is stateless), but there is still the follow up message causing a 2:1 amplification. I think something like the NTP interleaved mode would be better.
It seems they didn't replace the PTP offset calculation assuming a constant delay (broadcast model). That doesn't work well when the distribution of the delay is not symmetric, e.g. errors in hardware timestamping on the NIC are sensitive to network load. They would need to measure the actual error of the clock to see that (the graphs in the article seem to show only the offset measured by SPTP itself, a common issue when improvements in time synchronization are demonstrated).
I think a better solution taking advantage of existing PTP support in hardware is to encapsulate NTP messages in PTP packets. NICs and switches/routers see PTP packets, so they provide highly accurate timestamps and corrections, but the measurements and their processing can be full-featured NTP, keeping all its advantages like resiliency and security. There is an IETF draft specifying that:
An experimental support for NTP-over-PTP is included in the latest chrony release. In my tests with switches that work as one-step transparent clocks the accuracy is same as with PTP (linuxptp).
I listened to a fascinating podcast from Jane Street on their solution.
Pasted the relevant part
https://signalsandthreads.com/clock-synchronization/
--quote
"So, we’re trying to build a proof of concept. At the end of the day, we sort of figured, “All right. We have these GPS appliances.” We talked about hardware timestamping before on the GPS appliances, and how they can’t hardware timestamp the NTP packets, so that’s problematic. We thought, “How can we move time from the GPS appliances off into the rest of the network?” And so we decided that we could use PTP to move time from the GPS appliances to a set of Linux machines, and then on those Linux machines we could leverage things, like hardware timestamping, and the NTP interleaved mode to move the time from those machines onto machines further downstream.
The NTP interleaved mode, just to give a short overview of what that means… when you send a packet if you get it hardware timestamps on transmission the way you use that hardware timestamp is you get it kind of looped back to you as an application. So I transmit a packet, I get that hardware timestamp after the packet’s already gone out the door. That’s not super useful from an NTP point of view, because really you wanted the other side to receive that hardware timestamp, and so the interleaved mode is sort of a special way in which you can run NTP and that when you transmit your next NTP packet you send that hardware timestamp that you got for the previous transmission, and then the other side, each side can use those. I don’t want to get into too much of the details of how that works, but it allows you to get more accuracy and to leverage those hardware timestamps on transmission."
-- end quote
SPTP does look a lot like NTP over PTP. I’m guessing they deployed this last year when nothing like it existed - the ietf draft (dated Jan 24, 2024) came much later after. They might even be involved with it. Anyways, it’s nice to see progress towards a simpler protocol that retains the precision of PTP.
> TIL there's a regular heartbeat in the quantum foam; there's a regular monotonic heartbeat in the quantum Rydberg wave packet interference; and that should be useful for distributed applications with and without vector clocks and an initial time synchronization service
> A big problem with the PTP unicast mode is an almost infinite traffic amplification (useful for DDoS attacks). The server is basically a programmable packet generator. Never expose unicast PTP to internet. In SPTP that seems to be no longer the case (the server is stateless), but there is still the follow up message causing a 2:1 amplification. I think something like the NTP interleaved mode would be better.
Facebook has little concern for traffic amplification that doesn't affect them. I can't find a source article for it now, but there was a time when you could take down a website hosting an image by simply posting <URL>/?<RANDOM>. I believe Facebook's (many) cache servers would individually make requests to the server until they inevitably saturated the image host's connection. I remember people complaining and it falling on deaf ears.
but this is not about facebook, this discussion is about the protocol
given how this industry protocols work, is likely that other big corporations that run data centers are also part of the real protocol discussion, some of those will be corncerned about traffic amplification
An advantage of constantly loading at least one core of the CPU might be preventing the deeper power states from kicking in, which should make the RX timestamping latency more stable and improve stability of synchronization of NTP clients.