CCNP Tshoot - Advanced Services Troubleshooting (AND, Netflow, IP SLA, NBAR, QOS, Wireless)

 - flow is a series of packets, all of which share header information such as source and destination IP addresses, protocols numbers...

 - Large enterprise networks carry multiple types of application traffic for their users.

Application Network Services Troubleshooting
 - Cisco Application Network Services (ANS) is a collection of Cisco solutions that fall under the Cisco Service-Oriented Network Architecture (SONA) framework.
 - ANS technologies can enhance the performance of applications within a data center, for users at a remote site, and for a teleworker ,
 - Cisco offers a collection of ANS devices that can help optimize the performance of network-based applications.
 - In addition to dedicated hardware, however, Cisco IOS Software offers a collection of features that can help collect baseline information for these applications and then help improve application performance (for example, response time).

ANS Network Components
 - Cisco Application Velocity System (AVS) - Enhances web applications (for example, by measuring response time and by managing application layer security)
 - Cisco Global Site Selector (GSS)  - Optimizes distributed data center environments
 - Cisco Content Switching Module (CSM) - Performs load balancing across multiple devices (such as servers or firewalls)
 - Cisco Application Control Engine (ACE) - Performs intelligent load balancing and content switching to increase application availability
 - Cisco Wide Area Application Engine (WAAE)  - Provides a platform on which users can run Cisco ACNS or Cisco WAAS software
 - Cisco Wide Area Application Software (WAAS)  - Accelerates applications for remote office workers
 - Cisco Application and Content Networking System (ACNS) - Supports content distribution (for example, video streaming) to remote sites over an IP WAN

The performance of network applications can be enhanced by gaining an understanding of the application traffic, followed by optimizing the network for those applications.
This application optimization process can be summarized:
1) Baseline - The first step is to baseline the performance metrics of existing application traffic.
2) Optimize - After you understand the current behavior of the application traffic, you can optimize identified applications (for example, using QoS mechanisms).
3) Monitor - After implementing your optimization configuration, network traffic should again be monitored to determine how network traffic patterns are impacted by the new configuration.
4) Deploy - As a network evolves, new applications might be added, while existing applications might undergo multiple upgrades. Because the deployments of new applications or upgrades can affect the behavior of network applications, these steps should be repeated.

 - The Cisco IOS NetFlow feature can be used when baselining network application performance.
 - flow is a series of packets, all of which share header information such as source and destination IP addresses, protocols numbers, port numbers, and Type of Service (TOS) field information.
 - NetFlow can keep track of the number of packets and bytes observed in each flow and this info is stored in a flow cache
 - flow cache of a router can be exported to a NetFlow collector prior to the entries expiring.
 - NetFlow collector can produce reports detailing traffic statistics.

NetFlow Sample Configuration
R4# conf term
R4(config)# int fa 0/0
R4(config-if)# ip flow ingress
R4(config)# int fa 0/1
R4(config-if)# ip flow ingress

R4(config)# ip flow-export source lo 0
R4(config)# ip flow-export version 5
R4(config)# ip flow-export destination 5000
NetFlow - Flow Cache Check:
R4# show ip cache flow
IP packet size distribution (171856M total packets):
   1-32   64   96  128  160  192  224  256  288  320  352  384  416  448  480
   .000 .700 .061 .067 .004 .000 .001 .002 .000 .008 .005 .000 .035 .000 .071

    512  544  576 1024 1536 2048 2560 3072 3584 4096 4608
   .000 .000 .000 .000 .039 .000 .000 .000 .000 .000 .000

Protocol         Total    Flows   Packets Bytes  Packets Active(Sec) Idle(Sec)
--------         Flows     /Sec     /Flow  /Pkt     /Sec     /Flow     /Flow
TCP-Telnet       82118      0.0         2    57      0.0       6.8      20.7
TCP-FTP          63484      0.0         1    44      0.0       1.9      21.5
TCP-FTPD         28830      0.0         1    43      0.0       0.1      14.8
TCP-WWW        9164285      2.1       144   199    309.3       5.8      21.4
TCP-SMTP   25278350647   5885.5         4    52  27322.7       3.3      21.7
TCP-X           716507      0.1         1    40      0.1       2.0      23.4

If NetFlow is not behaving as expected, consider the following list of common NetFlow troubleshooting targets that could be investigated:
■ No network connectivity exists between a NetFlow router and its configured NetFlow collector
■ The router’s NetFlow configuration is incorrect
■ The NetFlow collector’s configuration is incorrect
■ An ACL or a firewall is blocking NetFlow traffic

 - Cisco IOS IP SLA feature  - can be used to measure how the network treats traffic for specific applications
 - IP SLA accomplishes this by synthetically generating traffic bearing similar characteristics to application traffic
 - This traffic, called probes, is sent to a destination router. This destination router is configured to respond to the received probes with time-stamp information, which can then be used to calculate performance metrics for the traffic.
 - IP SLAs can be used for baselining network application performance.

IP SLA Source Configuration
ip sla monitor 1
type tcpConnect dest-ipaddr dest-port 80 source-port 17406
tos 64
ip sla monitor schedule 1 life forever start-time now
IP SLA Responder Configuration
ip sla monitor responder
ip sla monitor responder type tcpConnect ipaddress port 80
Viewing Information Collected on the IP SLA Source
R1# show ip sla monitor statistics
Round trip time (RTT) Index 1
Latest RTT: 168 ms
Latest operation start time: *16:10:52.453 UTC Sun Mar 3 2002
Latest operation return code: OK
Number of successes: 13
Number of failures: 1
Operation time to live: Forever
BB2# show ip sla monitor responder
IP SLA Monitor Responder is: Enabled
Number of control message received: 15 Number of errors: 1
Recent sources: [00:38:01.807 UTC Fri Mar 1 2002] [00:37:01.783 UTC Fri Mar 1 2002]
Recent error sources: [00:24:01.807 UTC Fri Mar 1 2002] RTT_FAIL
tcpConnect Responder:
    IP Address Port
Common IP SLA issue
 - The source or destination IP addresses configured on the IP SLA source or responder is incorrect.
 - The frequency of the probes is set for a value that is too long.
 - The schedule is set to begin sending probes at some time in the future.
 - The probes are being filtered by an ACL or a firewall.

NBAR - Network-Based Application Recognition
 - NBAR can classify various traffic types by examining information at Layers 3 through 7.
 - Protocols that change port numbers (that is, stateful protocols) can also be tracked.
 - Although Cisco IOS comes with multiple NBAR application signatures, there is a continuing need for additional signature recognition capabilities.
 -  NBAR signature recognition capability can be expanded by adding one or more PDLM files to a router’s flash.
   (Example: you can install a Bit Torrent Packet Description Language Module (PDLM) into the router’s flash)
 - NBAR can function as a protocol discovery tool. Therefore, like NetFlow and IP SLA, NBAR can serve as a useful baselining tool.
 - The protocol discovery feature of NBAR can be enabled on an interface to determine the applications consuming the most bandwidth on that interface (that is, the top talkers).

NBAR Configuration
Router(config-if)# ip nbar protocol-discovery   <-- enable NBAR protocol discovery
Router(config)# ip nbar pdlm <pdlm_file>        <-- expanding NBAR Recognition by adding one or more PDLM files to a router’s flash.
Router(config)# ip nbar port-map protocol {tcp udp} <port_number> [<port_number>]  <-- modify the ports the NBAR uses
NBAR Verification
R1#sh ip nbar protocol-discovery     <-- NBAR collected traffic statistics for an interface
 Last clearing of "show ip nbar protocol-discovery" counters 00:04:38
                            Input                    Output
                            -----                    ------
   Protocol                 Packet Count             Packet Count
                            Byte Count               Byte Count
                            5min Bit Rate (bps)      5min Bit Rate (bps)
                            5min Max Bit Rate (bps)  5min Max Bit Rate (bps)
   ------------------------ ------------------------ ------------------------
   icmp                     2030                     1015
                            231420                   115710
                            0                        0
                            18000                    9000
   bgp                      0                        0
                            0                        0

Common NBAR troubleshooting issues:
■ NBAR is not correctly recognizing applications
 - This can occur if an application is using a nonstandard port.
 - use the show ip nbar port-map protocol command to see what port(s) is associated with a specified application.
 - For example, perhaps a web server is using TCP port 8080 instead of port 80.
 - issue the show ip nbar port-map http command and see that NBAR is only recognizing TCP port 80 as HTTP traffic.
 - use the ip nbar port-map http tcp 80 8080 command in global configuration mode to cause NBAR to recognize either TCP port 80 or 8080 as HTTP traffic.
■ NBAR does not support a specific application
 - If a PDLM file exists for an application, you can download it from, copy it to the flash of a router, and reference it with the
   ip nbar pdlm <pdlm-file> command in global configuration mode.
■ NBAR degrades router performance
 - Depending on the underlying router platform, the performance of a router might suffer as a result of NBAR’s inspection of multiple flows.
 - You can use the show processes cpu command to determine the CPU utilization of a router.

 - Modular QoS CLI (MQC)
 - AutoQoS Enterprise

AutoQoS Enterprise is configured via a three-step process:
R4(config)# int fa0/0
R4(config-if)# auto discovery qos    <---  AutoQoS Enterprise discovery phase
R4# show auto discovery qos    <--- After the discovery phase runs for a period of time (at least two or three days)
AutoQoS Discovery enabled for applications
Discovery up time: 1 minutes, 7 seconds
AutoQoS Class information:
Class Voice:
Recommended Minimum Bandwidth: 5 Kbps/<1% (PeakRate)
Detected applications and data:
Application/ AverageRate PeakRate Total
Protocol (kbps/%) (kbps/%) (bytes)
----------- ----------- -------- ------------
rtp audio 1/<1 5/<1 10138
Suggested AutoQoS Policy for the current uptime:
class-map match-any AutoQoS-Voice-Fa0/0
match protocol rtp audio
policy-map AutoQoS-Policy-Fa0/0
class AutoQoS-Voice-Fa0/0
priority percent 1
set dscp ef
class class-default
R4(config)# int fa 0/0
R4(config-if)# auto qos    <-- Apply the recommended policy using the auto qos
Common reasons that AutoQoS (both AutoQoS VoIP and AutoQoS Enterprise) might not function correctly on a router:
■ Cisco Express Forwarding (CEF) is not enabled
 - AutoQoS can use NBAR to recognize traffic types, and NBAR requires CEF to be enabled. You can enable CEF with the ip cef

■ An interface’s bandwidth is not correctly configured
 - Cisco IOS assumes the bandwidth of a serial interface to be 1.544 Mbps (that is, the bandwidth of a T1 circuit).
 - Because serial interfaces often run at different speeds, you should configure interfaces bandwidth <bandwidth-in-kbps>
 - Some routing protocols (for example, EIGRP and OSPF) can reference this bandwidth amount when calculating a route metric.
 - AutoQoS can reference this bandwidth amount to determine which QoS mechanisms should be enabled on an interface.
 - If the bandwidth value of an interface is left at its default setting, AutoQoS might not optimally configure QoS on an interface.

■ An interface has not been configured with an IP address
 - One QoS mechanism that AutoQoS might configure is Multilink PPP (MLP), which is a link fragmentation and interleaving mechanism.
 - Part of an MLP configuration involves the creation of a virtual multilink interface that needs to have an IP address assigned.
 - AutoQoS takes the needed IP address from the physical interface being configured for AutoQoS.
 - An interface should be configured with an IP address prior to configuring the interface for AutoQoS.

■ Only one side of a link has been configured
 - AutoQoS is enabled on an interface.
 - However, the interface in the router at the other end of the link needs a complementary configuration.
 - For example, consider two routers interconnected via a serial link running at a link speed of 512 kbps. If you configured AutoQoS for the interface at one end of the link, that interface might be configured for QoS mechanisms that include MLP and RTP header compression (cRTP). However, these mechanisms will not function correctly until the interface at the other end of the link is similarly configured.

■ The configuration created by AutoQoS has been modified
 - AutoQoS configurations are based on QoS mechanisms available in Cisco IOS.
 - Therefore, a configuration generated by AutoQoS can be customized.
 - As a result, the underlying issue you are troubleshooting might be caused by the customization of AutoQoS’ configuration

Wireless Troubleshooting Targets
 -  CCNP R/S focuses on troubleshooting targets on a wired network that could impact a wireless network.

Introducing the Cisco Unified Wireless Network
 - Wireless local-area networks (WLAN) offer network access via radio waves.
 - Wireless clients (such as PCs or PDAs) access a WAP using half-duplex communication.
 - The WAP allows a wireless client to communicate with the wired portion of a network.

Components comprise the Cisco Unified Wireless Network architecture:
■ Wireless clients - A wireless client device is typically an end-user device (such as a PC) that accesses a wireless network.
■ WAP - WAPs offer network access for wireless clients.
■ Wireless network unification - wireless network needs to be integrated (that is, unified) with a wired LAN.
■ Wireless network management - a wireless LAN can use network management solutions to enhance security and reliability and offer assistance in WLAN deployments (ex: Cisco Wireless Control System (WCS).
■ Wireless Mobility - include security threat detection, voice services, location services, and guest access.

 - Traditional WLANs use an access point in autonomous mode, where the access point is configured with a service set identifier (SSID), radio frequency (RF) channel, and RF power settings.
 - Aside from autonomous mode, Cisco Unified Wireless Networks can alternatively operate in split-MAC mode (WAP is considered a LWAP - lightweight access point - which cannot function without a WLC).

 - WLAN client sending traffic to the wired LAN sends a packet to a lightweight access point, which encapsulates the packet using Lightweight Access Point Protocol (LWAPP).
 - The encapsulated traffic is sent over an LWAPP tunnel to a WLC.
 - LWAPP sends packets in a Layer 2 frame with an Ethertype of 0xBBBB.
 - LWAPP data traffic uses a UDP destination port of 12222, whereas LWAPP control traffic uses a UDP destination port of 12223.

A lightweight access point - performs functions such as beaconing, packet transmission, and frame queuing.
WLC  - assumes roles such as authentication, key management, and resource reservation.

 - WLC uses an Extensible Authentication Protocol (EAP) to communicate with the authentication server.
 - Cisco Secure Access Control Server (ACS) can, for example, act as an authentication server.

Supported EAP types include the following:
■ EAP-Transport Layer Security (EAP-TLS)
 - Wireless clients and authentication servers mutually authenticate using digital certificates.
■ EAP-Protected EAP (EAP-PEAP)
 - The authentication server (that is, a RADIUS server) is authenticated over a Transport Layer Security (TLS) tunnel using a digital certificate,
 - Wireless clients are authenticated via Extensible Authentication Protocol—Generic Token Card (EAP-GTC) or Extensible Authentication Protocol—
Microsoft Challenge Handshake Authentication Protocol version 2 (EAP-MSCHAPv2).
■ EAP Tunneled Transport Layer Security (EAP-TTLS)
 - The RADIUS server is authenticated over a TLS tunnel using the certificate of the server, and wireless clients authenticate using username and password.
■ Cisco Lightweight Extensible Authentication Protocol (LEAP)
 - Cisco developed LEAP as an early and proprietary EAP method; however, LEAP’s vulnerability to a dictionary attack represents a major LEAP weakness.
■ Cisco EAP-Flexible Authentication via Secure Tunneling (EAP-FAST)
 - Cisco proposed EAP-FAST to address weaknesses of LEAP.

WLAN controller components:
■ Ports - A port on a WLAN controller physically connects the WLAN controller to the wired network (for example, to a Cisco Catalyst switch port).
■ Interfaces:An interface of a WLAN controller logically maps to a VLAN on a wired network.
■ WLANs:A WLAN can be configured with security features, QoS mechanisms, and other wireless network parameters. Also, a WLAN associates an SSID to a
WLC interface.

Wired Network Issues Impacting Wireless Networks
 - WAPs require power; however, these access points might need to be installed away from power outlets.
 - If the Cisco Catalyst switch is not appropriately configured to provide PoE, or if the switch has no additional power available, an attached WAP might fail to power on.
 - PoE troubleshooting aid is the Cisco Power Calculator, an online tool available at (can help determine the power capacity of a switch).

 - wireless users might experience connectivity issues if traffic from their wireless VLAN is not permitted over a trunk in the wired network.
 - trunk configurations on Cisco Catalyst switches might need to be inspected as part of troubleshooting wireless connectivity issues.
show vlan
show interfaces trunk
show interfaces switchport
 - ACLs might inadvertently be configured to block traffic that should be permitted on a network.
 - In the case of wireless networks configured in a splitMAC architecture, recall that UDP ports 12222 and 12223 should be permitted between a WAP and a WLC.

 - clients might need to roam from one subnet to another. In such an instance, a loss of wireless connectivity might result from a DHCP issue.

DHCP Troubleshooting Commands
show ip dhcp conflict        Lists any IP address conflicts identified by a router (via ping or gratuitous ARP)
show ip dhcp binding         Displays IP addresses assigned by an IOS DHCP server, MACs and lease expirations
clear ip dhcp binding *      Releases all current DHCP leases
clear ip dhcp conflict *     Clears all currently identified DHCP conflicts
debug ip dhcp server events  Provides real-time information about DHCP address assignments and database updates
debug ip dhcp server packet  Displays real-time decodes of DHCP packets
 - Latency-sensitive traffic traveling over a wireless network (for example, Voice over Wireless LAN [VoWLAN]) might suffer from poor performance if QoS markings are not preserved as traffic crosses the boundary between the wireless and wired portions of a network.
 - mls qos trust dscp - interface configuration mode command you can issue this command on a Cisco Catalyst switch to cause an interface to trust incoming DSCP markings and preserve priority markings on wireless traffic as it enters a wired network, you can issue this command on a Cisco Catalyst switch port that connects to a WLC.