Training E143 - Troubleshooting 800xA System\\E143doc

Expert Workshop E143

Ex ten ded A ut om at io n Sy st em 800x A    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Chapte hapt er 3 Trouble rou ble Shoo Shootin ting g Network

Network  Ar c h it ec ectt u r e

800xA Sys yste tem m Archi Arc hite tect cture ure Domain Server(s)

 Application Server(s)

Workplaces Client/Server Network

 Aspect Server(s)

Connectivity Server(s)

-  Single -  1oo2 -  2oo3

-  Single -  1oo2

Control Network Controllers

Fieldbus    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Field devices

Network  Ar c h it ec ectt u r e

Networ tw ork k Re Redund du nda ancy nc y

Primary client server net

Secondary client server net The The netw networ ork k cabl cables es shoul hould d be color lorcoded. coded. The The netwo network rk adapt adapters ers as well well..    3      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Primary control net Secondary control net

Network  Ar chit ecture

Redundant Ring Architecture

Secondary Network

Primary Network    4      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network  Ar chit ecture

Connectivity Servers as Routers

Enable TCP/IP forwarding must be set in RNRP configuration

   5      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

RNRP Modifies Routing Table RNRP adds the Connectivity Server in the routing table. Dynamically updated to path 1 if path 0 should fail.

Control Network destinations use Connectivity Server as gateway.

   6      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network  Ar chit ecture

Building Physical Networks

Logical View

   7      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Physical View

Network  Ar chit ecture

Physical Network Installation

Serial Structure

   8      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Tree Structu re Preferable when all nodes communicates equally with each other reduces the number or of hops

Network Troubleshooting

   9      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

RNRP Addressing

Network Troubleshooting

Implicit Addressing 1.

N1.N2.0.0 Choose a NetId which is conform through the whole 800xA net.

2.

255.255.252.0 This value must bet set for the sub net mask

Recommendations Use private Class B addresses: 172.16.0.0 or 172.20.0.0 or 172.24.0.0 or 172.28.0.0 172.16.0.0  less configuration efforts Use scheme in network manual for selection of addresses

   0    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Troubleshooting 800xA System

RNRP Re-routing to CS Lib

1.Client Helper  1.Shared Memory 1.SM Client

1.RNRP

   1    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

1.CS-Lib

Network Troubleshooting

   2    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

RNRP Event Monitor

“Left Click” on the icon

Network Troubleshooting

RNRP Fault Tracer 

Invoked by a right mouse double click

   3    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Use this to find any network errors and loops

Network Troubleshooting

RNRP Event Messages

 Add a pseudo node for each controller in order to get RNRP messages    4    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting





   5    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

RNRP Switch Considerations

Basic requirements 

Link speed should be possible to set by auto-sensing



Transmission mode, half or full duplex, should be possible to set by auto negotiation



Multicast traffic must be allowed



Port status should be visible on LEDs

Features not required 

IGMP Snooping can not be used and must be disabled



Multicast filtering must be disabled



Intelligent Multicast must be disabled

Network Troubleshooting





Troubleshooting Network Guidance

Understand the Problem 

Be aware of the network normal behavior 



Is it a problem?

Identify the problem 

 A sharp increase in utilization indicates an abnormal condition For example, a device might be causing "Broadcast Storms".



    6    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

 A sustained high or low level of utilization indicates an increasing or decreasing load on your network.

Solve the problem 

Narrow the utilization problem to the ports that have excessively high or low utilization.

Network Troubleshooting

Verify Network Settings Considerations Primary Client Server net

   7    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R



Refer to the domain controller with the Path 0 addresses



Register the Path 0 address in DNS

Network Troubleshooting

Verify Network settings Considerations Second ary Client Server n et

   8    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R



Refer to the domain controller with the Path 0 addresses. Note the opposite domain controller order for Path 1.



Don’t Register Path 1 connection in DNS

Network Troubleshooting

Verify Network Settings NetBIOS Enable NetBIOS for only for Primary Client/Server network Disable NetBIOS for all other Networks

   9    1      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

Verify Bindings Order  Primary Client/Server network must be in the first priority order  Check after reboot.

   0    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

   1    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Troubleshooting Tools



Hardware LEDs (Especially on Switches e.g. Collisions)



Different command-line tools: ping, pathping, tracert, netstat, netdiag



Large file copy technique



Task manager measurements



Performance Monitor measurements



WEB Interface on switches



From Microsoft Sysinternals: TCPView, TDIMon



Sniffer tools: Advanced IPScanner, Ethereal, Network chemistry packetyzer 

Network Troubleshooting

Command-line Tools Ping can be used with several options: • name  check how name is resolved • -l buffer size  check response times with some load • -a resolves address to name • -t continues ping

Tracert shows the path to the remote system.    2    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

RNRP Influences Ping Command Pinging a physical address may not be a verification that specific NIC, switch is up and running. • Pinging a Path 0 address  response from 172.16.4.11 • Primary Client/Server net switch is shut down • Still response from Path 0 172.16.4.11

   3    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

Command-line Tools Pathping traces the path to a remote computer and report packet losses along the way.

   4    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

The example shows a packet loss between the Connectivity server and the controller.

Network Troubleshooting

Command-line Tools Use Netstat to find out statistic about the net. • -a connection and listening ports • -e Ethernet statistics • -s statistics per protocol IP, UDP, TCP etc.

   5    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

Command-line Tools Netdiag checks the network connections. The windows support tool must be installed. Use the correct version, either Windows 2000, or Windows Server 2003.

   6    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

Dos Commands summary

Utility

   7    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Used to

Arp

View the ARP (Address Resolution Protocol) cache on the interface of the local computer to detect invalid entries.

Hostname

Display the host name of the computer.

Ipconfig

Display current TCP/IP network configuration values, and update or release Dynamic Host Configuration Protocol (DHCP) allocated leases, and display, register, or flush Domain Name System (DNS) names.

Nbtstat

Check the state of current NetBIOS over TCP/IP connections, update the NetBIOS name cache, and determine the registered names and scope.

Netstat

Display statistics for current TCP/IP connections.

Netdiag

Check all aspects of the network connection.

Nslookup

Check records, domain host aliases, domain host services, and operating system information by querying Internet domain name servers.

Pathping

Trace a path to a remote system and report packet losses at each router along the way.

Ping

Send ICMP Echo Requests to verify that TCP/IP is configured correctly and that a remote TCP/IP system is available.

Route

Display the IP routing table, and add or delete IP routes.

Tracert

Trace a path to a remote system.

Network Troubleshooting





   8    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Test Throughput by Large File Copy

On a 100 MBit network deduct 20% for synchronization divide the rest with 8 and you get 10 Mbytes/s.  As a rule of thumb a 50 MByte file should take approximately 5 sec.



With this technique you can check that speed and duplex setting is correct on NIC card and switch.



You will notice dramatically changes if some settings are wrong.

Network Troubleshooting

Networking in Task Manager  The Task Manager can be used to check the network utilization on-line when specific actions are taken e.g. Display changes or to check if there is a constant high load.

 As general guidelines, your network is healthy in these conditions: • Utilization is running up to 15 percent most of the time. • Utilization is peaking at 30 to 35 percent for a few seconds at a time, with large gaps of time between peaks. • Utilization is peaking at 60 percent for a few seconds, with large gaps of time between peaks.

   9    2      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Link speed can not always be trusted

Network Troubleshooting

Performance Logs Call perfmon from the Run menu.

 Add a counter Select the Network interface as performance object Select the desired counter 

   0    3      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

Performance Logs

For long time troubleshooting open the Computer Management dialog from Administrative Tools or add the “Performance Logs and Alerts” snap-in to the Microsoft Management Console (MMC).  A counter log can only be created in the snap-in. The data is stored in a file which can later be read in performance management console.

   1    3      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

Sysinternals Tools (TDIMon)

TDIMon monitors activity at the Transport Driver Interface (TDI) level, TCP and UDP activities

RNRP “I am alive telegram”

   2    3      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

Sysinternals Tools (TCPView) Don’t show any unconnected endpoints

TDCPView shows network activities. Can be used to see if connections are deleted and build up again. Deleted connections are shown in red. New end points are shown in green.    3    3      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

Sniffer Tools Ethereal or WireShark Start of Capture Packets

Protocols can be analyzed after you have stopped the capture.

   4    3      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Network Troubleshooting

   5    3      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Conclusions – Most Common Causes



Cable problems such as bad connections or cable specifications not met



Connectivity Problems



DNS not configured properly



Mismatch between switch and NIC



Switch settings or interconnections incorrect



“Old” Firmware in switch



Mistakes in network architecture

 ABB Logo

   6    3      T    I    t    l   u   s   n   o   C    C  n   o    B   i   s    B   i    A  v   e    ©    R

Maintenenance Expert workshop

Exercis e 3.1 Trou bleshoo t network

3.1.1 Description This exercise deals with network trouble shooting using different DOS commands. The exercise emphasize on a faulty hardware connections and faulty DNS settings. The exercise shows also how managed switches can be used for diagnostic.

3.1.2 Objectives Upon completion of this exercise you will be able to: •

•

•

•

Understand how name resolution problem will cause bad performance in the network. Be able to use simple file copy to get an overview of the network  performance. Understand how to use DOS commands for network analyze. E.g. nslookup tracert, and pathping. Be able to use WEB server interfaces from managed switches and 3:rd party sniffer tools.

3.1.3 Reference documentation Microsoft Tech. net articles.

1/3

E143-03 Exercise 4.1 - RevD.doc

3.1.4 Exercise Steps 





2/3

You suspect network problems. Your task is to identify problems in the network configuration of TCP/IP. There are a number of problems planted in DCASCSx1. •

What do you find on Primary Client Server Net?

•

What do you find on secondary Client Server net?

•

What do you find on Control Net?

•

What more should you consider concerning the Network Settings?

Edit the host file and put an incorrect address in for one of the nodes. Try pinging that node by name. Try pinging it by address. What were the results this time? What is the  priority between DNS and host fille?

Eliminate the incorrect host file entries so that everything should be back to normal on all nodes. You suspect network performance problem. You will verify the  performance by copy a large file (at least 50 MB) from one node to the other. Is the throughput reasonable for the connection speed of your NIC? Copy in the other direction. Are the results the same? Test both network cards. Did you get the same result? What can be the reason