Key to maintaining and troubleshooting a network is the collection of information about that network.
Fortunately, Cisco IOS offers many commands that can be used for information gathering.
Mastery of these basic tools can dramatically reduce the time a troubleshooter spends isolating the specific information needed for a troubleshooting task.
Cisco IOS Diagnostic Tools
Cisco IOS offers multiple show commands useful for gathering information.
Include pipe - output to be filtered to only include lines that include the text
R1# show processes cpu
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 4 3 1333 0.00% 0.00% 0.00% 0 Chunk Manager
2 7245 1802 4020 0.08% 0.08% 0.08% 0 Load Meter
3 56 2040 27 0.00% 0.00% 0.00% 0 OSPF Hello 1
4 21998 1524 14434 0.00% 0.32% 0.25% 0 Check heaps
5 4 1 4000 0.00% 0.00% 0.00% 0 EDDRI_MAIN
...
R1# show processes cpu | include Check heaps
5 24710 1708 14467 1.14% 0.26% 0.24% 0 Check heaps
show ip interface brief | exclude unassignedBegin - skips the initial portion of the show running-configoutput and begins displaying the output where the routing protocol configuration begins
show ip route 172.16.1.0
show ip route 172.16.0.0 255.255.0.0 longer-prefixes
Redirecting show Command Output to a File
Redirecting Output to a TFTP Server
R1# show tech-support | redirect tftp://192.168.1.50/tshoot.txtRedirecting Output While Also Displaying the Output Onscreen
!
R1#
R1# show tech-support | tee tftp://192.168.1.50/tac.txtAppending Output to an Existing File
!
---------------------show version---------------------
Cisco IOS Software, C2600 Software (C2600-IPVOICE_IVS-M), Version 12.4(3b), RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Thu 08-Dec-05 17:35 by alnguyen
...OUTPUT OMITTED...
R1# show ip interface brief | append tftp://192.168.1.50/baseline.txt
!
R1#
Troubleshooting Connectivity
R1# ping 10.4.4.4The ping command does have several options that can prove useful during troubleshooting.
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.4.4.4, timeout is 2 seconds:
!!!!!
■ size:Specifies the number of bytes per datagram
■ repeat:Specifies the number of ICMP Echo messages sent (defaults to 5)
■ timeout:Specifies the number of seconds to wait for an ICMP Echo Reply
■ source:Specifies the source of the ICMP Echo datagrams
■ df-bit:Sets the do not fragmentbit in the ICMP Echo datagram
For example, if the pingresults indicate alternating failures and successes (that is, !.!.!), a troubleshooter might conclude that traffic is being load-balanced between the source and destination IP addresses. Traffic flowing across one path is successful, whereas traffic flowing over the other path is failing.
R1# ping 10.4.4.4 size 1500 repeat 9999 timeout 0- Perhaps you suspect that an interface has a nondefault maximum transmission unit (MTU) size.
- You could send ICMP Echo messages across that interface using the df-bit and size options of the ping command to specify the size of the datagram to be sent.
R1# ping 10.4.4.4 size 1500 df-bitM.M.M
- Notice the M in the ping responses.
- An M indicates that fragmentation was required but could not be performed because the do not fragment bit was set.
Therefore, you can conclude that a link between the source and destination is using a nonstandard MTU.
An extended ping can help
The extended ping feature allows you to granularly customize your pings.
R1# pingTelnet command is useful for troubleshooting Layer 4.
Protocol [ip]:
Target IP address: 10.4.4.4
Repeat count [5]: 1
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]: yes
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]: y
Sweep min size [36]: 1400
Sweep max size [18024]: 1500
Sweep interval [1]:
Type escape sequence to abort.
Sending 101, [1400..1500]-byte ICMP Echos to 10.4.4.4, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.
Success rate is 50 percent (51/101), round-trip min/avg/max = 60/125/232 ms
R1#telnet 192.168.1.50 80Troubleshooting Hardware
Trying 192.168.1.50, 80 ... Open
show processes cpu
Provides 5-second, 1-minute, and 5-minute CPU utilization statistics, in addition to a listing of processes running on a platform along with
each process’s utilization statistics
show memory
Displays summary information about processor and I/O memory, followed by a more comprehensive report of memory utilization
show interfaces
Shows Layer 1 and Layer 2 interface status, interface load information, and error statistics including the following:
■ input queue drops:Indicates a router received information faster than the information could be processed by the router
■ output queue drops:Indicates a router received information faster than the information could be sent out of the outgoing interface (perhaps because of an input/output speed mismatch)
■ input errors:Indicates frames were not received correctly (for example, a cyclic redundancy check (CRC) error occurred), perhaps indicating a cabling problem or a duplex mismatch
■ output errors:Indicates frames were not transmitted correctly, perhaps due to a duplex mismatch
NOTE: Prior to collecting statistics, interface counters can be reset using the clear counterscommand.
show controllers
Displays statistical information for an interface (for example, error statistics), where the information varies for different interface types (for example, the type of connected cable might be displayed for a serial interface)
show platform
Provides detailed information about a router or switch hardware platform
Specialized Diagnostic Tools
Cisco offers network maintenance and troubleshooting tools targeted toward this type of data collection.
Most of the information collection tasks performed in the previous processes fall into one of three categories:
■ Troubleshooting information collection: This information collection is conducted as part of troubleshooting a reported problem.
■ Baseline information collection: This information collection is conducted when the network is operating normally, to form a frame of reference against which other data can be compared.
■ Network event information collection: Some network devices can be configured to automatically generate alerts in response to specific conditions (for example, configured utilization levels on a switch, router, or server being exceeded).
Performing Packet Captures
You can use dedicated appliances or PCs running packet capture software to collect and store packets flowing across a network link.
You can also look inside Layer 2, 3, and 4 headers using a packet capture application.
Capturing and analyzing packets, however, presents two major obstacles. First, the volume of data collected as part of a packet capture can be so large that finding what you are looking for can be a challenge. Therefore, you should understand how to use your packet
capture application’s filtering features.
A second challenge is that if you want to monitor, for example, traffic flow between two network devices connected to a switch, the packets traveling between those two devices will not appear on your packet capture device’s switch port.
Cisco IOS supports a feature known as SPAN - instructs a switch to send copies of packets seen on one port (or one VLAN) to another port.
Cat3550(config)# monitor session 1 source interface gig 0/1Remote SPAN(RSPAN) - network capture device connected to one switch might need to capture packets flowing through a different switch.
Cat3550(config)# monitor session 1 destination interface gig 0/3
Cat3550(config)# end
Cat3550# show monitor
Session 1
------------Type : Local Session
Source Ports :
Both : Gi0/1
Destination Ports : Gi0/3
Encapsulation : Native
Ingress : Disabled
SW1(config)# vlan 20SW2 config:
SW1(config-vlan)# name SPAN
SW1(config-vlan)# remote-span
SW1(config-vlan)# exit
SW1(config)# monitor session 1 source interface gig 0/1
SW1(config)# monitor session 1 destination remote vlan 20 reflector-port gig 0/3
SW1(config)# end
SW1#show monitor
Session 1
------------
Type : Remote Source Session
Source Ports :
Both : Gi0/1
Reflector Port : Gi0/3
Dest RSPAN VLAN : 20
SW2(config)# vlan 20
SW2(config-vlan)# name SPAN
SW2(config-vlan)# remote-span
SW2(config-vlan)# exit
SW2(config)# monitor session 2 source remote vlan 20
SW2(config)# monitor session 2 destination interface fa 5/2
SW2(config)# end
SW2# show monitor
Session 2
------------
Type : Remote Destination Session
Source RSPAN VLAN : 20
Destination Ports : Fa5/2
Creating a Baseline with SNMP and NetFlow
Simple Network Management Protocol (SNMP) and NetFlow are two technologies available on some Cisco IOS platforms that can automate the collection statistics.
SNMP
Collects device statistics (for example, platform resource utilization, traffic counts, and error counts)
Uses a pull model (that is, statistics pulled from monitored device by a network management station [NMS])
Available on nearly all enterprise network devices
R1(config)# snmp-server community CISCO ro
R1(config)# snmp-server community PRESS rw
R1(config)# snmp-server contact demo@ciscopress.local
R1(config)# snmp-server location 3rd Floor of Wallace Building
R1(config)# snmp-server ifindex persist
NetFlow
Collects detailed information about traffic flows.
Uses a push model (that is, statistics pushed from the monitored device to a NetFlow collector).
Available on routers and high-end switches.
Unlike SNMP, NetFlow can distinguish between different traffic flows.
A flow is a series of packets, all of which have shared header information such as source and destination IP addresses, protocols numbers, port numbers, and Type of Service (TOS) field information.
NetFlow can keep track of the number of packets and bytes observed in each flow.
This information is stored in a flow cache.
You can use the NetFlow feature as a standalone feature on an individual router.
However, rather than using just a standalone implementation of NetFlow, entries in a router’s flow cache can be exported to a NetFlow
collectorprior to the entries expiring.
R4(config)# int fa 0/0Although these protocols by themselves lack a mechanism to alert a network administrator (for example, via e-mail) when a network event is logged, third party software is available that can selectively alert appropriate personnel when specific events are logged.
R4(config-if)# ip flow ingress <---enables NetFlow for that interface
R4(config)# int fa 0/1
R4(config-if)# ip flow ingress
R4(config)# ip flow-export source lo 0 <---specifies the interface used to communicate with an external NetFlow collector
R4(config)# ip flow-export version 5 <---specifies the NetFlow version used by a device
R4(config)# ip flow-export destination 192.168.1.50 5000 <---specifies the IP address and port number of an external NetFlow collector
Notifications that a managed device sends to an NMS are called traps.
R4# show ip cache flowFileds description - http://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/ios-netflow/prod_white_paper0900aecd80406232.html
IP packet size distribution (168365M total packets):
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .699 .060 .067 .004 .000 .001 .002 .000 .008 .005 .000 .035 .000 .073
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .000 .000 .000 .039 .000 .000 .000 .000 .000 .000
IP Flow Switching Cache, 4456704 bytes
65197 active, 339 inactive, 2237763651 added
3832429833 ager polls, 0 flow alloc failures
Active flows timeout in 5 minutes
Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 2630408 bytes
65197 active, 16723 inactive, 2237763622 added, 2237763622 added to flow
0 alloc failures, 0 force free
5 chunks, 21 chunks added
last clearing of statistics never
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)
-------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow
TCP-Telnet 75181 0.0 2 58 0.0 7.0 20.5
TCP-FTP 58954 0.0 1 44 0.0 2.0 21.5
TCP-FTPD 25921 0.0 1 44 0.0 0.1 14.8
TCP-WWW 8905939 2.0 148 196 307.0 5.7 21.3
TCP-SMTP 24709693864 5753.1 4 52 26771.8 3.3 21.5
TCP-X 692434 0.1 1 40 0.1 2.0 23.2
TCP-BGP 885564 0.2 2 48 0.4 12.5 20.5
TCP-NNTP 3908 0.0 1 40 0.0 1.9 22.9
TCP-Frag 6692686 1.5 7 418 12.2 7.5 22.5
TCP-other 28028372 6.5 29 369 193.8 5.2 20.0
UDP-DNS 16798907814 3911.3 1 95 4066.5 0.0 23.1
UDP-NTP 33530200 7.8 23 246 182.6 0.9 27.2
UDP-TFTP 330 0.0 71 466 0.0 0.5 24.9
UDP-Frag 1365 0.0 1 1377 0.0 2.2 23.5
UDP-other 7838864523 1825.1 3 650 6257.3 0.0 23.0
ICMP 55799847 12.9 108 68 1406.5 5.3 22.5
IGMP 5 0.0 1 28 0.0 0.8 27.6
IPINIP 6 0.0 2 80 0.0 9.0 20.2
IPv6INIP 26 0.0 5 70 0.0 6.2 24.4
IP-other 129771 0.0 62 61 1.9 277.6 4.7
Total: 49482296710 11521.0 3 156 39200.6 1.7 22.3
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Gi0/1.12 192.94.94.27 Gi0/3.10 1.1.1.1 11 BC0D 0035 1
Gi0/1.11 2.2.2.2 Gi0/3.11 65.55.37.72 06 7409 0019 4
Providing Notifications for Network Events
Rather than enabling all possible traps with the snmp-server enable traps command,
you can selectively enable traps using the snmp-server enable traps <trap name> command.
R4(config)# snmp-server host 192.168.1.150 version 2c CISCOPRESSEEM
R4(config)# snmp-server enable traps
% Cannot enable both sham-link state-change interface traps.
% New sham link interface trap not enabled.
R4# show run | include traps
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps vrrp
snmp-server enable traps eigrp
snmp-server enable traps xgcp
snmp-server enable traps ds3
...OUTPUT OMITTED...
Cisco IOS supports a featured called EEM that allows you to create your own event definitions and specify custom responses to those events.
An event can be defined based on syslog messages, SNMP traps, and entering specific Cisco IOS commands, as just a few examples.
In response to a defined event, EEM can perform various actions, including sending an SNMP trap to an NMS, writing a log message to a syslog server, executing specified Cisco IOS commands, sending an e-mail to an appropriate party, or executing a tool command language (Tcl) script.
R4(config)# event manager applet COUNTER-RESET- Note that the clear counters command would be detected even if a shortcut (for example, cle co) were used.
R4(config-applet)# event cli pattern “clear counters” sync no skip no occurs 1
R4(config-applet)# action A syslog priority informational msg “Please update network documentation to record why the counters were reset.”
- The sync no parameter says that the EEM policy will run asynchronously with the CLI command. The skip no parameter says that the CLI command will not be skipped (that is, the CLI command will be executed).
- the occurs 1parameter indicates that the EEM event is triggered by a single occurrence of the clear counters command being issued
R4# clear counters
Clear “show interface” counters on all interfaces [confirm]
R4#
*Mar 3 08:41:00.582: %HA_EM-6-LOG: COUNTER-RESET: Please update network documentation to record why the counters were reset.