In recent years the utility market has been moving away from serial and towards Ethernet as its communication network of choice. Using Ethernet, interoperability between various vendors is much easier, as the lower levels of Ethernet call for standards of how data packets are constructed and sent across the network. Different applications will often still use proprietary protocols at the higher levels; however the way the data is transmitted on the network is the same.
The industrial market has been following the same trend, also moving over to Ethernet, although the industrial move has been a lot easier due to there generally being less remote legacy devices in the industrial world, which means less devices that have to be replaced or converted for compatibility with an Ethernet network. The industrial sector is also more forgiving when arranging scheduled downtime for installation of new hardware or expansion of the network.
Serial connections have many limitations, including distance, susceptibility to EMI (Electro-Magnetic Interference), slow transmission speeds and of course lack of interoperability between different vendors and protocols. Ethernet introduces many new protocols and mediums of communication to lessen or negate these limitations.
Serial allows for a maximum distance of around 1.2km (Depending on many factors, such as environment, protocol, speeds required etc.) Longer distances means lower speeds, as well as being more affected by EMI etc. meaning planning must be done to make sure that the required speeds and data reliability are achievable, and that the connection will be stable.
Ethernet extends this distance almost indefinitely (Depending on application, and locations of end points etc.) by providing many different mediums. The medium used will depend on the application, and what is required. For instance over short distances (Such as within a patch panel/control room) a Cat5e (Category 5 Extended) or Cat6 cable would be used. Fiber optics can be used for longer distances, up to about 3km for 100Mbps Multimode fiber and up to 90km or longer for 100Mbps Singlemode fiber. If distances of even greater than this are required, options are available for dedicated satellite links, or for less mission critical applications cellular routers can be used to connect to the Internet, and from there data can effectively be sent anywhere in the world that has Internet connectivity.
Due to serial communications mostly taking place over copper based wiring, these links are highly susceptible to EMI. In the utility environment this is especially a concern due to the high voltages and EMI produced. Ethernet can lessen the impact of EMI by using fiber optic cabling as opposed to copper cabling, as fiber optics are completely immune to EMI. Although the actual Ethernet devices themselves can still be affected by EMI, some manufacturers do eliminate this by providing specially shielded products that provide resistance or even immunity to EMI.
Serial communications are extremely limited on their speeds, or baud rates (Symbols per second). Depending on the hardware and software speeds can reach up to around 1Mbps. However these speeds drop dramatically as we increase the distance of the serial link. However, even over an extremely short link (<5m) the maximum achievable speed of around 1Mbps is nowhere near to the speeds provided by Ethernet, which allows speeds of up to 10 Gbps (Or even higher if we aggregate multiple links together). As serial is mainly used for communications between a small number of devices with very small amounts of data these low speeds were not a problem. However Ethernet can be used to create networks covering a whole substation and, with the correct planning and configuration, can be used to send data between all devices on the site. With this allowance for so much more data than that provided by serial, devices can now send and receive that much more data, allowing for more advanced protocols and device management.
As stated, Ethernet also allows for much better interoperability between devices. The reason for this is that Ethernet is an open standard, and devices that support Ethernet must support this standard (IEEE 802.3). This means that different manufacturers’ Ethernet devices will always be able to transmit and receive data between one another (Although whether they can understand the data depends on the protocol being used). Different manufacturers may use proprietary higher level protocols, however these protocols will still be packeted and sent on the network with the same specifications as a different manufacturers protocols. This means that a single Ethernet network can be constructed to cater for whatever data is required. However more planning than serial is required for creating an Ethernet network, as details such as required data rates, number of devices per switch/device and so on are required.
However, Ethernet networks allow for much easier expansion than serial does. By simply adding new switches to the existing network, more ports and thus more devices can be catered for. If a higher bandwidth backbone is required due to increased traffic amount, multiple links can be aggregated together to provide the required bandwidth.
Another downfall in serial is the lack of redundancy provided. Although some devices will come with a redundant serial port, generally serial devices only have the single port. This means that any cable breaks can potentially cause devices to become unreachable, which can lead to serious problems, downtime or in extreme cases even death if protection devices fail. Ethernet caters for many different kinds of redundancy. Using protocols such as RSTP (Rapid Spanning Tree Protocol) redundancy can be provided, up to a level required by the application. This redundancy means that in the case of a cable break, traffic flow can be redirected to allow all devices to still be reachable.
HSR (High-availability Seamless Redundancy) and PRP (Parallel Redundancy Protocol) are two newer redundancy protocols that provide redundancy in different ways.
HSR works by sending the same data in two directions around a ring or mesh network. The receiving node on the other end of the network will simply discard the data that arrives second. However in the event that there is a break in the network, one set of data will still be able to get through to the receiving device.
PRP goes so far as to run two completely separate Ethernet networks redundant to one another. This means that even in the event that one network catastrophically fails the second network will be able to pick up the load almost instantaneously, meaning no data is lost in the changeover.
Ethernet also allows for much more control over the traffic flow on a network. Using protocols such as VLANs (Virtual Local Area Networks) we can segregate data based on which group of devices the data needs to be received by. For instance we could have control, protection and surveillance all running on different VLANs, meaning that devices in the control VLAN would not be affected by traffic in the protection or surveillance VLANs and vice versa. When using VLANs it is important to make sure that the network backbone can cater for the total amount of traffic of all VLANs combined.
We can then go even further by combining redundancy along with VLANs to achieve load balancing on the backbone using a technology called MSTP (Multiple Spanning Tree Protocol). MSTP provides redundancy using RSTP, however rather than having the same redundant link for all VLANs (Which then means that link is unused except in cases where there is a cable break) MSTP provides a different redundant link for different VLANs. For instance VLAN 1 could be using a link that VLAN 2 is not using, and vice versa. MSTP therefore allows for much greater control over the traffic flow on the network backbone, whilst still providing the high-speed recovery provided by RSTP in the event of a cable or hardware failure.
Figure 1 – Example of an MSTP Configured Network
Generally Ethernet is not considered deterministic, as due to the number of devices and variable traffic flow on a basic Ethernet network it is difficult if not impossible to determine a reliable time frame in which packets will transfer between devices. However with the correct planning and configuration certain traffic on the network can be made deterministic.
Using IEEE1588v2 aka PTPv2 (Precision Time Protocol v2) all PTP enabled devices on the network can be synchronized to within 1 µs (microsecond) of a master clock. This allows for much more accurate timing and monitoring of the transmission speeds of the packets. This alone will not make a network deterministic; however it is the first step in getting deterministic traffic. PTP is extremely valuable for applications that require precise time measurements, such as synchrophasor readings.
A second technology that, in conjunction with PTP, can add to the determinism of a network, is prioritization. This works by assigning different priorities to different Ethernet packets. A packet that has not already been assigned a priority will be assigned one based on the first of the following inspections that has been configured and enabled by the user:
- By the ToS field in the IP header – This prioritization must be assigned by a layer 3 device, however it can be inspected by a layer 2 device (A switch).
- Priority field in the 802.1Q tag – Once again this field must be assigned by a layer 3 device, however a switch can inspect this field and prioritize the packet accordingly.
- Source or destination MAC address – The switch will check its static MAC table and assign a priority to the packet if so configured.
- By ingress port – The switch will assign the packet a priority based on the port it enters the switch on.
Figure 2 – Prioritisation Flow Chart
There are two methods to determine how to deal with prioritized packets. Most manufacturers will offer both of these methods in their devices, however some manufacturers will only support one or the other. Each of these methods has it’s own pros and cons.
The Strict or Starve method of handling prioritization works by first transmitting all higher priority frames before moving on to the low priority ones. The pro to this method in that all high priority packets are giving the most attention, ensuring that all high priority packets are sent ASAP. The con if of course that if there is too many high priority packets then lower priority packets can potentially never be transmitted.
The Weighted Fair Queuing method involves setting a queuing configuration (Normally 8:4:2:1) for sending the packets in a fair ratio. This setting will send up to (It can send less if there are not enough packets of that priority, i.e. it will not hold off sending data until all queues are full) 8 critical priority packets, 4 high priority, 2 med priority and 1 low priority packet at a time. The pro to this is that all data will have a chance to be sent, no matter what the traffic amounts are like, however the con is that critical data could potentially be delayed depending on the rest of the traffic on the network.
Using these different protocols certain critical data travelling on the network can be made to have deterministic like behaviour, in that we will be able to determine a time frame wherein the packet must be transmitted from one device to another.
So we can see that the overall benefits and ease of use provided by Ethernet far outweigh those provided by serial. Ethernet is definitely the next logical choice for all industrial, utility and ITS networks, and with the proper understanding and configuration can be an extremely powerful technology for the transmission of data. With the redundancy, time synchronization, prioritization and other protocols available, Ethernet can reliably be used for critical data networks and extremely time sensitive applications. Due to the open standards, many different vendors Ethernet based hardware can all be combined on the same network, saving on costs for cabling, hardware etc. Ethernet is definitely the network of choice for the present and, due to its popularity and expansion on a world-wide scale, it can be expected to stay that way for the foreseeable future. For any more information on Industrial, Utility or ITS Ethernet hardware and software contact H3iSquared.
Tim Craven, H3iSquared
Tel: +27 (0)11 454 6025
Email: info@h3isquared.com