CCNP Tshoot - The Maintenance and Troubleshooting Toolbox


Key to maintaining and troubleshooting a network is the collection of information about that network.
Fortunately, Cisco IOS offers many commands that can be used for information gathering.
Mastery of these basic tools can dramatically reduce the time a troubleshooter spends isolating the specific information needed for a troubleshooting task.

Cisco IOS Diagnostic Tools
Cisco IOS offers multiple show commands useful for gathering information.

Include pipe - output to be filtered to only include lines that include the text
R1# show processes cpu
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 4 3 1333 0.00% 0.00% 0.00% 0 Chunk Manager
2 7245 1802 4020 0.08% 0.08% 0.08% 0 Load Meter
3 56 2040 27 0.00% 0.00% 0.00% 0 OSPF Hello 1
4 21998 1524 14434 0.00% 0.32% 0.25% 0 Check heaps
5 4 1 4000 0.00% 0.00% 0.00% 0 EDDRI_MAIN
...
R1# show processes cpu | include Check heaps
5 24710 1708 14467 1.14% 0.26% 0.24% 0 Check heaps
show ip interface brief | exclude unassigned
Begin -  skips the initial portion of the show running-configoutput and begins displaying the output where the routing protocol configuration begins

show ip route 172.16.1.0
show ip route 172.16.0.0 255.255.0.0 longer-prefixes

Redirecting show Command Output to a File
Redirecting Output to a TFTP Server
R1# show tech-support | redirect tftp://192.168.1.50/tshoot.txt
!
R1#
Redirecting Output While Also Displaying the Output Onscreen
R1# show tech-support | tee tftp://192.168.1.50/tac.txt
!
---------------------show version---------------------
Cisco IOS Software, C2600 Software (C2600-IPVOICE_IVS-M), Version 12.4(3b), RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Thu 08-Dec-05 17:35 by alnguyen
...OUTPUT OMITTED...
Appending Output to an Existing File
R1# show ip interface brief | append tftp://192.168.1.50/baseline.txt
!
R1#

Troubleshooting Connectivity

R1# ping 10.4.4.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.4.4.4, timeout is 2 seconds:
!!!!!
The ping command does have several options that can prove useful during troubleshooting.
■ size:Specifies the number of bytes per datagram
■ repeat:Specifies the number of ICMP Echo messages sent (defaults to 5)
■ timeout:Specifies the number of seconds to wait for an ICMP Echo Reply
■ source:Specifies the source of the ICMP Echo datagrams
■ df-bit:Sets the do not fragmentbit in the ICMP Echo datagram

For example, if the pingresults indicate alternating failures and successes (that is, !.!.!), a troubleshooter might conclude that traffic is being load-balanced between the source and destination IP addresses. Traffic flowing across one path is successful, whereas traffic flowing over the other path is failing.
R1# ping 10.4.4.4 size 1500 repeat 9999 timeout 0
 - Perhaps you suspect that an interface has a nondefault maximum transmission unit (MTU) size.
 - You could send ICMP Echo messages across that interface using the df-bit and size options of the ping command to specify the size of the datagram to be sent.
R1# ping 10.4.4.4 size 1500 df-bit
M.M.M
 - Notice the M in the ping responses.
 - An M indicates that fragmentation was required but could not be performed because the do not fragment bit was set.
Therefore, you can conclude that a link between the source and destination is using a nonstandard MTU.

An extended ping can help
The extended ping feature allows you to granularly customize your pings.
R1# ping
Protocol [ip]:
Target IP address: 10.4.4.4
Repeat count [5]: 1
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]:
Set DF bit in IP header? [no]: yes
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]: y
Sweep min size [36]: 1400
Sweep max size [18024]: 1500
Sweep interval [1]:
Type escape sequence to abort.
Sending 101, [1400..1500]-byte ICMP Echos to 10.4.4.4, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.M.
Success rate is 50 percent (51/101), round-trip min/avg/max = 60/125/232 ms
Telnet command is useful for troubleshooting Layer 4.
R1#telnet 192.168.1.50 80
Trying 192.168.1.50, 80 ... Open
Troubleshooting Hardware
show processes cpu
Provides 5-second, 1-minute, and 5-minute CPU utilization statistics, in addition to a listing of processes running on a platform along with
each process’s utilization statistics

show memory
Displays summary information about processor and I/O memory, followed by a more comprehensive report of memory utilization

show interfaces
Shows Layer 1 and Layer 2 interface status, interface load information, and error statistics including the following:
input queue drops:Indicates a router received information faster than the information could be processed by the router
output queue drops:Indicates a router received information faster than the information could be sent out of the outgoing interface (perhaps because of an input/output speed mismatch)
input errors:Indicates frames were not received correctly (for example, a cyclic redundancy check (CRC) error occurred), perhaps indicating a cabling problem or a duplex mismatch
output errors:Indicates frames were not transmitted correctly, perhaps due to a duplex mismatch
NOTE: Prior to collecting statistics, interface counters can be reset using the clear counterscommand.

show controllers
Displays statistical information for an interface (for example, error statistics), where the information varies for different interface types (for example, the type of connected cable might be displayed for a serial interface)

show platform
Provides detailed information about a router or switch hardware platform

Specialized Diagnostic Tools
Cisco offers network maintenance and troubleshooting tools targeted toward this type of data collection.

Most of the information collection tasks performed in the previous processes fall into one of three categories:
■ Troubleshooting information collection: This information collection is conducted as part of troubleshooting a reported problem.
■ Baseline information collection: This information collection is conducted when the network is operating normally, to form a frame of reference against which other data can be compared.
■ Network event information collection: Some network devices can be configured to automatically generate alerts in response to specific conditions (for example, configured utilization levels on a switch, router, or server being exceeded).

Performing Packet Captures
You can use dedicated appliances or PCs running packet capture software to collect and store packets flowing across a network link.
You can also look inside Layer 2, 3, and 4 headers using a packet capture application.

Capturing and analyzing packets, however, presents two major obstacles. First, the volume of data collected as part of a packet capture can be so large that finding what you are looking for can be a challenge. Therefore, you should understand how to use your packet
capture application’s filtering features.

A second challenge is that if you want to monitor, for example, traffic flow between two network devices connected to a switch, the packets traveling between those two devices will not appear on your packet capture device’s switch port.


Cisco IOS supports a feature known as SPAN - instructs a switch to send copies of packets seen on one port (or one VLAN) to another port.
Cat3550(config)# monitor session 1 source interface gig 0/1
Cat3550(config)# monitor session 1 destination interface gig 0/3
Cat3550(config)# end
Cat3550# show monitor
Session 1
------------Type : Local Session
Source Ports :
Both : Gi0/1
Destination Ports : Gi0/3
Encapsulation : Native
Ingress : Disabled
Remote SPAN(RSPAN) - network capture device connected to one switch might need to capture packets flowing through a different switch.
SW1(config)# vlan 20
SW1(config-vlan)# name SPAN
SW1(config-vlan)# remote-span
SW1(config-vlan)# exit
SW1(config)# monitor session 1 source interface gig 0/1
SW1(config)# monitor session 1 destination remote vlan 20 reflector-port gig 0/3
SW1(config)# end
SW1#show monitor
Session 1
------------
Type : Remote Source Session
Source Ports :
Both : Gi0/1
Reflector Port : Gi0/3
Dest RSPAN VLAN : 20
SW2 config:
SW2(config)# vlan 20
SW2(config-vlan)# name SPAN
SW2(config-vlan)# remote-span
SW2(config-vlan)# exit
SW2(config)# monitor session 2 source remote vlan 20
SW2(config)# monitor session 2 destination interface fa 5/2
SW2(config)# end
SW2# show monitor
Session 2
------------
Type : Remote Destination Session
Source RSPAN VLAN : 20
Destination Ports : Fa5/2 

Creating a Baseline with SNMP and NetFlow
Simple Network Management Protocol (SNMP) and NetFlow are two technologies available on some Cisco IOS platforms that can automate the collection statistics.

SNMP
Collects device statistics (for example, platform resource utilization, traffic counts, and error counts)
Uses a pull model (that is, statistics pulled from monitored device by a network management station [NMS])
Available on nearly all enterprise network devices
R1(config)# snmp-server community CISCO ro
R1(config)# snmp-server community PRESS rw
R1(config)# snmp-server contact demo@ciscopress.local
R1(config)# snmp-server location 3rd Floor of Wallace Building
R1(config)# snmp-server ifindex persist

NetFlow
Collects detailed information about traffic flows.
Uses a push model (that is, statistics pushed from the monitored device to a NetFlow collector).
Available on routers and high-end switches.

Unlike SNMP, NetFlow can distinguish between different traffic flows.
A flow is a series of packets, all of which have shared header information such as source and destination IP addresses, protocols numbers, port numbers, and Type of Service (TOS) field information.
NetFlow can keep track of the number of packets and bytes observed in each flow.

This information is stored in a flow cache.

You can use the NetFlow feature as a standalone feature on an individual router.

However, rather than using just a standalone implementation of NetFlow, entries in a router’s flow cache can be exported to a NetFlow
collectorprior to the entries expiring.
R4(config)# int fa 0/0
R4(config-if)# ip flow ingress   <---enables NetFlow for that interface
R4(config)# int fa 0/1
R4(config-if)# ip flow ingress

R4(config)# ip flow-export source lo 0    <---specifies the interface used to communicate with an external NetFlow collector
R4(config)# ip flow-export version 5     <---specifies the NetFlow version used by a device
R4(config)# ip flow-export destination 192.168.1.50 5000    <---specifies the IP address and port number of an external NetFlow collector
Although these protocols by themselves lack a mechanism to alert a network administrator (for example, via e-mail) when a network event is logged, third party software is available that can selectively alert appropriate personnel when specific events are logged.
Notifications that a managed device sends to an NMS are called traps.
R4# show ip cache flow
IP packet size distribution (168365M total packets):
   1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
   .000 .699 .060 .067 .004 .000 .001 .002 .000 .008 .005 .000 .035 .000 .073

    512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
   .000 .000 .000 .000 .039 .000 .000 .000 .000 .000 .000

IP Flow Switching Cache, 4456704 bytes
  65197 active, 339 inactive, 2237763651 added
  3832429833 ager polls, 0 flow alloc failures
  Active flows timeout in 5 minutes
  Inactive flows timeout in 15 seconds
IP Sub Flow Cache, 2630408 bytes
  65197 active, 16723 inactive, 2237763622 added, 2237763622 added to flow
  0 alloc failures, 0 force free
  5 chunks, 21 chunks added
  last clearing of statistics never
Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet       75181      0.0         2    58      0.0       7.0      20.5
TCP-FTP          58954      0.0         1    44      0.0       2.0      21.5
TCP-FTPD         25921      0.0         1    44      0.0       0.1      14.8
TCP-WWW        8905939      2.0       148   196    307.0       5.7      21.3
TCP-SMTP   24709693864   5753.1         4    52  26771.8       3.3      21.5
TCP-X           692434      0.1         1    40      0.1       2.0      23.2
TCP-BGP         885564      0.2         2    48      0.4      12.5      20.5
TCP-NNTP          3908      0.0         1    40      0.0       1.9      22.9
TCP-Frag       6692686      1.5         7   418     12.2       7.5      22.5
TCP-other     28028372      6.5        29   369    193.8       5.2      20.0
UDP-DNS    16798907814   3911.3         1    95   4066.5       0.0      23.1
UDP-NTP       33530200      7.8        23   246    182.6       0.9      27.2
UDP-TFTP           330      0.0        71   466      0.0       0.5      24.9
UDP-Frag          1365      0.0         1  1377      0.0       2.2      23.5
UDP-other   7838864523   1825.1         3   650   6257.3       0.0      23.0
ICMP          55799847     12.9       108    68   1406.5       5.3      22.5
IGMP                 5      0.0         1    28      0.0       0.8      27.6
IPINIP               6      0.0         2    80      0.0       9.0      20.2
IPv6INIP            26      0.0         5    70      0.0       6.2      24.4
IP-other        129771      0.0        62    61      1.9     277.6       4.7
Total:     49482296710  11521.0         3   156  39200.6       1.7      22.3

SrcIf         SrcIPaddress    DstIf         DstIPaddress    Pr SrcP DstP  Pkts
Gi0/1.12      192.94.94.27    Gi0/3.10      1.1.1.1     11 BC0D 0035     1
Gi0/1.11      2.2.2.2   Gi0/3.11      65.55.37.72     06 7409 0019     4
Fileds description - http://www.cisco.com/c/en/us/products/collateral/ios-nx-os-software/ios-netflow/prod_white_paper0900aecd80406232.html

Providing Notifications for Network Events
Rather than enabling all possible traps with the snmp-server enable traps command,
you can selectively enable traps using the snmp-server enable traps <trap name> command.
R4(config)# snmp-server host 192.168.1.150 version 2c CISCOPRESS
R4(config)# snmp-server enable traps
% Cannot enable both sham-link state-change interface traps.
% New sham link interface trap not enabled.
R4# show run | include traps
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps vrrp
snmp-server enable traps eigrp
snmp-server enable traps xgcp
snmp-server enable traps ds3
...OUTPUT OMITTED...
EEM
Cisco IOS supports a featured called EEM that allows you to create your own event definitions and specify custom responses to those events.
An event can be defined based on syslog messages, SNMP traps, and entering specific Cisco IOS commands, as just a few examples. 
In response to a defined event, EEM can perform various actions, including sending an SNMP trap to an NMS, writing a log message to a syslog server, executing specified Cisco IOS commands, sending an e-mail to an appropriate party, or executing a tool command language (Tcl) script.
R4(config)# event manager applet COUNTER-RESET
R4(config-applet)# event cli pattern “clear counters” sync no skip no occurs 1
R4(config-applet)# action A syslog priority informational msg “Please update network documentation to record why the counters were reset.”
 - Note that the clear counters command would be detected even if a shortcut (for example, cle co) were used.
 - The sync no parameter says that the EEM policy will run asynchronously with the CLI command. The skip no parameter says that the CLI command will not be skipped (that is, the CLI command will be executed).
- the occurs 1parameter indicates that the EEM event is triggered by a single occurrence of the clear counters command being issued
R4# clear counters
Clear “show interface” counters on all interfaces [confirm]
R4#
*Mar 3 08:41:00.582: %HA_EM-6-LOG: COUNTER-RESET: Please update network documentation to record why the counters were reset.