Troubleshooting Industrial Ethernet Networks
Unlike office environments, slowdowns in industrial networks can quickly escalate into production nightmares. You definitely need some help.
January 1, 2009
Print this page
Ethernet is rapidly spreading across a wide range of industrial environments, but as frontline engineers are quickly realizing the hostile environmental conditions can play havoc with their networks.
This standardized, plug-and-play technology is easy to install and it works 99% of the time, but when it does not, it is not so easy to understand why.
With more devices being added to the network and fewer people on the factory floor, production and maintenance engineers are under growing pressure to ensure 24/7 network uptime. If they don’t, the consequences can be devastating.
Unlike office environments, slowdowns in industrial networks can quickly escalate into production nightmares. As a result, production and maintenance staff are under increasing pressure to keep networks up and running at top speed, and when they’re not, resolving any issues as quickly as possible in order to maintain or improve productivity and uptime.
In an industrial network, you may encounter cases of sensors in the network sending their information to multiple addresses instead of transmitting to the single, relevant address.
A pressure sensor in a bottling plant for example could be sending measurements to labeling, and capping in addition to the bottling machine, resulting in unexpected network traffic.
In addition, if the bottling line is not correctly segmented from regular network traffic, it may send the bottling-machine pressure information (a time-critical packet of information) at the same time as dozens of people in the corporate network are busily downloading a new network security fix, causing major ramifications on the production line.
On the industrial side there is also much greater sensitivity to delay or latency than in office environments.
When a forecast file takes 20 seconds longer to download than usual, a sales manager will not be too concerned. In the bottling line, the difference between 25 milliseconds and 2000 milliseconds could result in a run of bottles without labels — and all the costs that go with that kind of mistake.
This article explores some of the basic design and installation issues for industrial networks, as well as troubleshooting tips to help you quickly recognize, locate, and solve problems in order to maintain or improve productivity and plant uptime.
Design and installation: As with any network, taking care of the basics when it comes to network design and installation can play a key role in improving overall long-term performance.
There are several things to consider when installing structured cabling in an industrial setting, these include:
• Design to standards: Make sure to adhere to the TIA 568A installation guidelines (standards from the Telecommunications Industry Association that apply to structured cabling). When installing STP (shielded twisted pair) /fiber in a noisy and harsh environment, pay close attention to the cable length maximums and the quality of terminations.
• Understand the performance characteristics needed in your specific production environment: In most cases, they tend to require much lower bandwidth than office settings. (Sending 2K of a communication command requesting a temperature reading is more critical than a receiving a complete 2 MB photo and being able to see it clearly). It is more important to focus on accommodating multiple data commands traversing a process system, rather than prioritizing bandwidth or network transmission speed.
• Understand your communication priorities: With industrial Ethernet it is important to ensure that smaller commands continuously reach their end at exactly the right time. Cat 5e transmission requirements are usually sufficient to handle this type of traffic. Also gain a thorough understanding of the electrical noise generated by any nearby machinery when deciding upon UTP (unshielded twisted pair), STP or fiber optic components. *
• Use the right components: Use protective cabling in any areas with repetitive motion or an area that a link might be stepped on or crushed. Many cable manufactures offer specific cabling with boots and/or advanced engineered material cable jackets specifically for harsh environments.
The basic troubleshooting sequence for any networking environment consists of these steps:
Document the network — Create a diagram of the parts of the network, as well as the signal flow paths between various devices, using for example the EtherScope Network Assistant from Fluke Networks for discovery. This is enormously helpful in visualizing the problem and saving valuable time. A good practice is to start with the physical layer and work your way up the protocol stack (OSI model).
Collect all available information and analyze the symptoms — Verify or recreate the problem where possible.
Localize and isolate the problems — A good overall troubleshooting strategy is to divide and conquer; test and eliminate. Subdivide or isolate the problems into a smaller function sections to eliminate culprits one by one, removing the largest and/or most convenient section first.
The step-by-step process should include:
• Pinging or simulating signals to check for power and proper connections
• If the connections are okay, proceeding to the control or management device
• Looking at the PLC for error codes
• Document what you did — As an important final step in troubleshooting, documenting of all the steps you took will provide a baseline which will be helpful in troubleshooting similar problems, prepare reports or train network support team members.
No matter what your cable type, it is important that you test and certify each link. Following are some of the more common issues you are likely to encounter when troubleshooting your network.
• Contaminated fiber optic cable end faces — Microscopic fiber optic cable is particularly susceptible to contaminants clogging the ends leading to communication breakdowns. This will show up as errors in the transmitted data and could be catastrophic.
• Device installation errors — A common and difficult-to-troubleshoot issue is misconfigured devices. The original installer must select the speed of the connection between the switch and sensor. In some cases, the sensor may be configured for half duplex, and the switch on the other end to full duplex in error. Duplex mismatch is very difficult to detect since standard testing will indicate that the cabling is fine and that both the switch and sensor are connected. The only clue will be extremely variable delay or latency so the network appears slow. A good tester can tell you the configuration of each device and will help you locate a mismatch.
• Device failures caused by a harsh electrical environment — Electrical signals carried over the network are susceptible to the harsh electrical and climatic environment. Several factors can influence or degrade electrical signaling, including high-energy sources such as lighting systems and heavy-duty machinery. Monitoring the network, one segment at a time can capture these events so you can isolate their source.
• High-voltage electrical loads — Transients, surges, and harmonics are the most common electrical phenomena found on high-voltage electrical feeder and branch circuits with breakers and non-linear loads. These and other phenomena like static discharge from rotating machinery can cause disturbances to electrical signals. Ground-loop currents are also notorious for creating erroneous equipment failures that are hard to diagnose. With attention to proper installation practices, minimizing sources of electrical disturbances, and using appropriately IP-rated devices you can mitigate many of these problems.
Although similar principles apply to installations in a data centre, Ethernet installations in dirty and noisy industrial environments require special consideration. To ensure your installation meet
s the customer and environmental needs you must design to standards, use the right components, and understand the appropriate troubleshooting techniques and equipment. As Ethernet continues to be deployed across industrial environments, new and specialized test tools are available to help isolate industrial Ethernet problems quickly and easily. CNS
David Green, P. Eng., is Director of Marketing for Fluke’s AmPac Region, including Canada, Australia and LatinAmerica He can be reached by e-mail at firstname.lastname@example.org.