Data Communications
A strong, stable communication infrastructure is critical to everyone within the industrial and utility worlds. Data communication between different PLCs, RTUs, IEDs, servers, HMI machines etc. is very important as this is the backbone which allows for other several different types of communication. Visual communication involves remote monitoring of sites using video cameras, and the transmission of these images to/from local/remote areas. Audio communication allows people on the site to speak to one another over a data network, using a device such as a VoIP Telephone.
Previously these “networks” would have all been separate, with data running between devices of serial connections, video being pulled in to a central control room over point-to-point coaxial cables, and audio often being provided by a combination of walkie talkies, leaky feeder systems, or a SMART/analogue telephone/intercom system. Recently, with the expansion of Ethernet as the network of choice, most industrial and utility sites have already started to move to an Ethernet based network. However many of these networks are not using Ethernet to its full potential, instead a very basic Ethernet network is installed to allow communication between devices. For this reason separate networks will often be installed for the video and audio communications so as to not overtax the data network, all of which can be avoided by correct, complete configurations for that application.
There are a few drawbacks to creating separate networks, most of which are quite obvious. One of the first drawbacks is the costs involved. When a second network needs to be installed for audio and visual communications the CAPEX will be effectively doubled, as separate cables, switches, routers etc. will be required. The same will be true of the OPEX costs, due to the fact that double the amount of spares will be required to effectively cater for problems on either network. A second drawback is that this will create a second network that will need to be monitored and maintained, which creates double the amount of work for the network administrators as well as providing double the amount of points of failure.
One of the main reasons that network engineers are often wary of including audio and visual communications on the critical data network is that the amounts of traffic generated can flood the network, causing delays or loss of critical data which can lead to downtime or incidents on the site. This can be overcome by using a combination of ruggedized switches with good switching fabrics (The amount of data the switches can handle internally simultaneously), routers that can handle the amount of traffic and network environments and the correct implementation of the following configurations: VLANs (Virtual LANS – IEEE 802.1Q); prioritization (IEEE 802.1p); and IGMP (Internet Group Management Protocol) for controlling multicasts on the network. VLANs should be used to separate various devices into separate logical groups. For instance PLCs could belong to VLAN 13 and cameras to VLAN 15. A VLAN will ensure that data from the PLCs will not be transmitted to any of the cameras and vice-versa. However the switches in the network will still need to deal with data from both VLAN 13 and VLAN 15. For this reason we use prioritization rules to tell the switches which is the critical data. Prioritization will ensure that critical data is given the highest priority, whilst still allowing lower priority data (Such as camera feeds) to still be transmitted. IGMP is used for ‘pruning’ multicast traffic on the network. Multicasts are data streams which will travel from 1 device to multiple devices which require the stream. For instance, consider a river that passes through 1 dam and then splits to 2 other dams, with the water being the data traffic. Smaller streams branching from each dam to outlying areas, which would simulate the end device connected to our switches (Dams). If only 1 outlying area requires water from a dam, a sluice gate will be opened to redirect water (Data) to that area. However the switches are still going to be receiving all the data, even if they have no end devices attached to them that require this multicast. IGMP prevents this by stopping data travelling to a switch that does not require it, thus decreasing the overall traffic travelling on the network backbone, but still allowing new members to receive the multicast at any time (IGMP performs dynamic configuration). Generally IGMP requires a IGMP capable router in the network to implement, however RuggedCom switches are able to be configured in an IGMP Active mode, meaning they will take the place of an IGMP router to control the multicasting on the network.
Another important aspect to always take into consideration on a critical network is redundancy. Redundancy is a method by which the network is ‘protected’ against cable break, loss of power or equipment failure, by allowing extra connections to be inactive (Redundant connections) until such a time as they are needed. As the majority of network problems are cable related, it is important to cater for this, so that a single cable breaking will not cause a network outage. Different Ethernet equipment vendors have different protocols for redundancy, but these are often proprietary, which means using these will vendor lock a company. RSTP (Rapid Spanning Tree Protocol – IEEE 802.1D) is one of the most commonly used forms of redundancy nowadays, due to it being a standard. However RSTP can only cater for up to 11 switches in a mesh topology, with recovery times of up to 30 seconds in a worst case scenario. This is a very large delay, especially on critical networks where loss of data can cause severe problems to arise.
For this reason RuggedCom developed eRSTP (Enhanced Rapid Spanning Tree Protocol). eRSTP improves upon RSTP by allowing up to 85 switches in a mesh topology with recovery times of up to 5ms per hop, which is a definite increase over RSTP. eRSTP is a proprietary protocol however it is still backwards compatible with RSTP and STP. So if an eRSTP enabled switch finds a non-eRSTP compatible switch ‘next to’ it, it will automatically drop only that link to RSTP, whilst keeping eRSTP on links that are compatible.
Another seldom used technology that provides a form of redundancy is Link Aggregation (IEEE 802.3ad). This protocol allows multiple links between the same two switches to be combined into a single logical link, and so is mainly used to increase bandwidth between backbone switches (If required). For instance, if we have 4 x 1000 Mbps links between two switches, we could combine these to act as a single 4 Gigabit link. If one of the physical cables were broken we would still have a 3 Gigabit link which would remain active and continue to transmit data. Although this does not provide network redundancy as e/R/STP does, it still provides link redundancy between the two switches, as well as providing the increased bandwidth where required.
So with the correct use and configuration of your switches and routers we can see that Ethernet can easily be tailored to cater for many different traffic types and priorities, whilst still ensuring that the critical data on the network is transmitted to the end device in a timely manner.
Video Communications
Once we have a stable, working Ethernet network in place it is time to start using that to provide vidual information from various locations. Previously video solutions used to include analogue cameras streaming their footage over co-axial cable to a central DVR (Digital Video Recorder), however this method has many disadvantages.
First off is the range limitation for this method. Running video over co-axial cable we can go up to about 500m when using a cable with no joins, or up to 1800m when using a booster to strengthen the signal. These are estimate figures, as a lot will depend on the type of cable being used, the equipment being used etc. Although these distances seem quite large they are still limiting the max range a camera can be installed from the DVR. Also it must be taken into consideration that these distances are for cable length, and due to the fact that cable installations will generally follow the contours of land and buildings, the effective distances from camera to DVR can be much less. Also, high EMI (Electro-Magnetic Interference) areas can cause data loss or corruption, leading to static and blurry/grainy images.
Another disadvantage of analogue cameras is the fact that each camera needs its own coaxial connection to the DVR. This means that if we have 4 cameras covering a remote substation 500m from the control room, we will need 4 cable runs to handle these cameras.
An IP camera can solve both of these problems. An IP based camera can be run over an existing Ethernet network (With the correct monitoring software and configuration in place) and so the effective distance can be extended to almost anywhere. With the Internet remote camera feeds can effectively be pulled in from anywhere in the world, using devices such as 3G routers for remote access to areas without existing Ethernet connectivity. On sites without internet connection options such as wireless or EoVDSL (Ethernet over VDSL lines) can be used to extend the Ethernet network to the required locations.
Multiple cameras can all be pulled over the same link, however again care must be taken to use the correct software and configuration so as not to overuse the available bandwidth, especially on the longer, lower bandwidth links. For instance, in the example above 4 camera feeds could all be run over a single 500m fibre pair, a quarter of the amount of cabling required when compared to an analogue camera.
The next thing to consider is the monitoring software that will be used to collect, store and supply the video feeds. Most camera monitoring software allows the cameras to ‘push’ their feeds to the software. This means that the cameras will always transmit at their highest bandwidth usage. This is one of the biggest reasons that users will often not want to use IP cameras on an existing network, as without prioritization these cameras could easily overwhelm the network with too much traffic causing critical data to be delayed or lost.
IV&C (Industrial Video and Control) has developed a camera monitoring software that solves the problem of bandwidth over-utilisation. IV&Cs RSS (Relay Server Software) works by ‘pulling’ the camera feeds from around the network, rather than the more traditional where the cameras ‘push’ their feeds. The software allows the user to configure at what fps (frames per second) a camera feed will be streamed, meaning that bandwidth utilisation can be controlled and configured to an amount that will not overwhelm the network. So for instance, when an Engineer wants to check a camera to see whether a conveyor belt is moving, or that switchgear has correctly broken a circuit, we do not require a very high frame-rate. In these cases a frame-rate of 4 fps will generally suffice. As most IP cameras deliver on average between 10 and 25 fps of video footage, being able to drop this to only 1 fps means a large drop in network traffic. IV&C also allows bandwidth control based on whether the footage is for recording or viewing, and also between LAN or WAN recording. So for instance we could record at 10 fps but only allow general viewers to view at 4 fps. This means that if ever needed, there is footage recorded at a higher framerate for incident analysis etc. And we could set it so that a user viewing from the local LAN would be able to view at a higher framerate than a user viewing the footage via the Internet.
This bandwidth control also means that multiple cameras can even transmit their footage over a low bandwidth link such as 3G or wireless. Using this, we can situate remote cameras in highly remote areas and collect all the footage in a central control room.
Audio Communications
Now that we can effectively transmit data and video from remote sites, we need to consider being able to audibly contact people at the site. With cellphones or walkie-talkies, people in a control room can often contact engineers and technicians on a remote site, however these both have disadvantages.
First off is how stable is the connection. Some remote sites are in areas with little to no cellular reception. Also some of the sites have large amounts of interference, meaning connections can be erratic or even not possible. Walkie Talkies suffer from similar issues; they are limited by the maximum connection range, as well as by radio interference. Another problem is background noise. In high noise environments it is often hard or impossible to make out what is being said over a cellular phones or walkie talkie connection.
Another option is to use a standard analogue phone system, however this introduces its own problems, one of the largest of which is cabling. An analogue phone system requires dedicated analogue links between each phone and the switchboard, which can mean a large amount of cabling which could be avoided.
Another, potentially more feasible option to use in these cases is a VoIP (Voice over IP) telephone system. VoIP works over an Ethernet network, and does not use a high amount of bandwidth. For this reason it can be run over an existing Ethernet network without causing problems, although again it is recommended to configure prioritization so that voice traffic is given relatively high priority, but not higher than the critical control data travelling the network.
GAI-Tronics supplies a line of ITS (Intelligent Traffic Solutions), industrial and utility grade phones for use in harsh environments. With a wide variety of options such as IP (Ingress Protection) rated phones and enclosures, intrinsically safe ratings, “clean” phones for medical environments and many more, GAI-Tronics phones can be customised to the users and environments requirements. These phones come in 3 different connection types; Standard analogue; Standard VoIP and SMART (Self-Monitoring And Reporting Technology). SMART phones connect to a standard analogue phone whilst still allowing a PC with a special modem connected to “interrogate” the phones to discover their current running status, any errors they have detected (Such as a stuck button or handset not correctly hung up) as well as to get statistical information such as last numbers dialled or number of calls made in a day.
GAI-Tronics phones can be customised in their physical buildup as well. Details such as the paint colour, the type of handset cable and the number of buttons can all be chosen.
Conclusion
So in conclusion we can see that communications are the backbone of the industrial, utility and military worlds, especially the data communications as the heart of the network. All IP communications rely on a strong, stable and reliable Ethernet network. The correct configuration and equipment is essential to creating an Ethernet network that is able to cater for not only critical data such as PLC, SCADA and control data, but can cater for audio, video and other applications on the network, as well as being able to expand easily in the future. All communications are significant when designing a safe, stable and secure network and all should be considered. Data, audio and visual do not need to be separated, and large amounts of budget can be saved by cutting down on cabling, spares and man hours when installing if a single, well configured Ethernet network is in place.
By Tim Craven
H3iSquared
Tel: +27 (0)11 454 6025
Email: info@h3isquared.com
Website: www.h3isquared.com