Beckhoff PTP (IEEE 1588) Troubleshooting
- Karl Newman
The EL6688 PTP cards used in the MMCS and CMCS have proven to be sensitive and problematic. These instructions are intended to help quickly diagnose and resolve issues with timing.
Symptoms
Both the MMCS and CMCS continuously receive trajectory commands from TCS via MCS. Included in the trajectory is a reference timestamp reflecting the current time known to TCS when the commands are sent. TCS receives its time updates via NTP from the GPS time servers, which is not as precise as PTP but provides a parallel path which can be useful for detecting differences.
A problem with timing will generally show up as a notable difference between the trajectory reference timestamp and the Beckhoff/TwinCat internal clock time. Typically this difference is about 0.05 to 0.1 seconds, due to various communication latencies. The difference appears in several different places:
On the MCS Engineering Screen, under the Time delta (s) column. The field will flash red if the difference is greater than ±0.5 seconds.
On the engineering interface for the MMCS/CMCS (typically running on the virtual machines used to access/program the PLCs). This can be seen by accessing the Configuration screen (button on lower left), on the Time tab. The Current Time Diff fields show the difference.
In the Ignition TMA Engineering project on the PTP Diag tab. The trjTimeDiff fields show the differences in time values:
Collecting Information
The MMCS and CMCS are monitored and programmed primarily through virtual machines which have been set up for this purpose. https://nso.atlassian.net/wiki/spaces/HLSDOCS/pages/2221834245 has details about how to access the virtual machines.
Launch TwinCAT System Manager. Ensure the correct target system is selected, and update it if necessary using Actions → Choose Target System… To get the most accurate configuration, choose to open from the target (File → Open from Target or click the red Open icon on the toolbar).
In the TwinCAT System Manager tree, navigate to I/O Devices → EtherCAT bus. Expand the first EK1100 bus coupler, locate the EL6688 card and select it. In the right hand side, select the CoE - Online tab. Click the Auto Update button to get continuously updated values.
At the bottom of the list, the PTP Diag contains some key values for diagnosing the health of the PTP card. The PTP Version, PTP State, and Timescale should always have the values shown above. The Grandmaster Identity should normally match the PTP identity of the primary GPS time server. The Offset from Master is typically a few hundred ns or less, but that is not a reliable indicator of the performance of the time system, because it has been observed to be working correctly with a large offset, and incorrectly with a small offset. The Sync Event Sequence Number should increase about once per second. Steps Removed is expected to remain at 3, absent some changes in the Ethernet infrastructure.
Above the PTP Diag, the Ethernet Settings should also be checked. Ensure that the Address Type is FIXED and that the IP Address and Subnetmask match the values assigned for that device on the PTP network. The Gateway should be 0, but it’s unused so the actual value is not important. Note that the values are in hex so each pair of hex digits will need to be converted to get a more conventional address. In the screenshot above, 0x0A010104 is 10.1.1.4, and the subnet mask of 0xFFFFFF00 is 255.255.255.0.
The PTPv2 Settings section should match the screenshot above, although most of the settings won’t actually break PTP but could affect its performance. The Priority1 and Priority2 are ignored because the setting in PTP Common is “Slave Only Clock” which means the card will never attempt to be the PTP master.
Back in the System Manager tree, expand I/O Devices and select the top EtherCAT node. Go to the EtherCAT tab and click the Advanced Settings… button.
The Distributed Clocks setting should show the EL6688 as the External Sync Device at least, and it will likely be used as the Reference Clock as well.
Troubleshooting
Reboot the Ethernet switch feeding PTP to the card; possibly upstream switches as well. Poorly behaving switches have caused problems in the past.
Restart the Beckhoff. Ideally use the reboot batch file on the desktop of the PLC’s embedded computer, instead of a hard power cycle or reset.
Shutdown the Beckhoff and remove power, then restart. Reference: Shutting down Beckhoff PLCs (Mount/Coudé) for maintenance
Try PTP from a different Ethernet switch. The GIS switch for TEL is in MCC2. A spare port can be configured for the PTP VLAN (301) and a temporary cable can be run from the TEL switch in MCC2 over to the PTP card in MCC1.
Swap in a spare EL6688 card.
Swap in a spare EK1100 EtherCAT bus coupler.
PTP Track Hound can be useful to check on the PTP traffic. The administrator of the switch can temporarily set up a port as a mirror of another port so the traffic going in and out of the PTP port can be observed.
Early on during testing and commissioning it was discovered that the EL6688 cards do not tolerate having traffic other than PTP on their Ethernet interface. More information here: https://www.dropbox.com/s/hlvlo2sdigvk4bj/Mount%20SAT%20Bills%20Notes.docx?dl=0
Related articles
Filter by label
There are no items with the selected labels at this time.