- The router accumulates NetFlow statistics in a NetFlow cache and can export them to an external device (such as the Cisco CNS NetFlow Collection Engine) for further processing.
- Large enterprise networks carry multiple types of application traffic for their users.
Application Network Services Troubleshooting
- Cisco Application Network Services (ANS) is a collection of Cisco solutions that fall under the Cisco Service-Oriented Network Architecture (SONA) framework.
- ANS technologies can enhance the performance of applications within a data center, for users at a remote site, and for a teleworker ,
- Cisco offers a collection of ANS devices that can help optimize the performance of network-based applications.
- In addition to dedicated hardware, however, Cisco IOS Software offers a collection of features that can help collect baseline information for these applications and then help improve application performance (for example, response time).
ANS Network Components
- Cisco Application Velocity System (AVS) - Enhances web applications (for example, by measuring response time and by managing application layer security)
- Cisco Global Site Selector (GSS) - Optimizes distributed data center environments
- Cisco Content Switching Module (CSM) - Performs load balancing across multiple devices (such as servers or firewalls)
- Cisco Application Control Engine (ACE) - Performs intelligent load balancing and content switching to increase application availability
- Cisco Wide Area Application Engine (WAAE) - Provides a platform on which users can run Cisco ACNS or Cisco WAAS software
- Cisco Wide Area Application Software (WAAS) - Accelerates applications for remote office workers
- Cisco Application and Content Networking System (ACNS) - Supports content distribution (for example, video streaming) to remote sites over an IP WAN
The performance of network applications can be enhanced by gaining an understanding of the application traffic, followed by optimizing the network for those applications.
This application optimization process can be summarized:
1) Baseline - The first step is to baseline the performance metrics of existing application traffic.
2) Optimize - After you understand the current behavior of the application traffic, you can optimize identified applications (for example, using QoS mechanisms).
3) Monitor - After implementing your optimization configuration, network traffic should again be monitored to determine how network traffic patterns are impacted by the new configuration.
4) Deploy - As a network evolves, new applications might be added, while existing applications might undergo multiple upgrades. Because the deployments of new applications or upgrades can affect the behavior of network applications, these steps should be repeated.
NetFlow
- https://www.cisco.com/en/US/technologies/tk648/tk362/technologies_white_paper09186a00800a3db9.html
- https://www.cisco.com/c/en/us/td/docs/ios/12_0s/feature/guide/nfexpfv9.html
- The Cisco IOS NetFlow feature can be used when baselining network application performance.
- flow is a series of packets, all of which share header information such as source and destination IP addresses, protocols numbers, port numbers, and Type of Service (TOS) field information.
- NetFlow can keep track of the number of packets and bytes observed in each flow and this info is stored in a flow cache
- flow cache of a router can be exported to a NetFlow collector prior to the entries expiring.
- NetFlow collector can produce reports detailing traffic statistics.
- You must enable either Cisco Express Forwarding (CEF) or distributed CEF (dCEF) before flow-sampler-map, ip flow egress, ip flow ingress, netflow-sampler commands
- If your router is running Cisco IOS release 12.2(14)S or a later release, or Cisco IOS Release 12.2(15)T or a later release, NetFlow accounting might be enabled through the use of the ip flow ingress command instead of the ip route-cache flow command.
- How to Access the Data Produced by NetFlow?
1) CLI with show commands or utilizing an application reporting tool. If you are interested in an immediate view of what is happening in your network, the CLI can be used. NetFlow CLI is very useful for troubleshooting.
2) Export NetFlow to a reporting server or what is called the "NetFlow collector".
Active Flow Timeout
- Each cache has various configurable elements, such as the number of entries and the time that a flow is allowed to remain in it.
- When a flow times out, it is removed from the cache and sent to any exporters that are configured for the corresponding flow monitor.
- If a cache is already active (that is, you have applied the flow monitor to at least one interface in the router), your changes to the record, cache type, and cache size parameters will not take effect until you either reboot the router or remove the flow monitor from every interface and then reapply it.
- Whenever possible you should customize the record, cache type, and cache size parameters for the cache before you apply the flow monitor to an interface. You can modify the timers, flow exporters, and statistics parameters for a cache while the cache is active.
NetFlow (routers): ip flow-cache timeout active 1
NetFlow (Catalyst switches): mls aging long 64
sFlow: polling interval 60
Netstream: ip netstream timeout active 1
J-Flow: ip flow-cache timeout active 1
Defaults:
The following flow cache parameters for a Flexible NetFlow flow monitor are enabled:
- Cache type: normal
- Maximum number of entries in the flow monitor cache: 4096
- Active flow timeout: 1800 seconds (30 min)
- Inactive flow timeout: 15 seconds
- Update timeout for a permanent flow cache: 1800 seconds (30 min)
NetFlow Versions
v1 First implementation, now obsolete, and restricted to IPv4 (without IP mask and AS Numbers).
v2 Cisco internal version, never released.
v3 Cisco internal version, never released.
v4 Cisco internal version, never released.
v5 Most common version, available (as of 2009) on many routers from different brands, but restricted to IPv4 flows.
v6 No longer supported by Cisco. Encapsulation information (?).
v7 Like version 5 with a source router field. Used (only?) on Cisco Catalyst switches.
v8 Several aggregation form, but only for information that is already present in version 5 records
v9 Template Based, available (as of 2009) on some recent routers. Mostly used to report flows like IPv6, MPLS, or even plain IPv4 with BGP nexthop.
v10 Used for identifying IPFIX. Although IPFIX is heavily based on NetFlow, v10 does not have anything to do with NetFlow.
- NetFlow version 9 is the IETF standard mechanism for information export. (link)
- The distinguishing feature of the NetFlow Version 9 format is that it is template based. Templates provide an extensible design to the record format, a feature that should allow future enhancements to NetFlow services without requiring concurrent changes to the basic flow-record format. Using templates provides several key benefits:
- The NetFlow protocol itself has been superseded by Internet Protocol Flow Information eXport (IPFIX).
- Based on the NetFlow Version 9 implementation, IPFIX is on the IETF standards track with RFC 5101 (obsoleted by RFC 7011), RFC 5102 (obsoleted by RFC 7012), etc. which were published in 2008.
- Version 9 is not backward-compatible with version 5 or version 8. If you need version 5 or version 8, then you must configure version 5 or version 8.
- Export bandwidth increases for version 9 (because of template flowsets) versus version 5.
- The increase in overhead versus version 5 varies with the frequency with which template flowsets are sent. With one template flowset sent per 10 export packets, the overhead is one percent versus version 5 export. With one template flowset sent for every export packet, the overhead is about eight percent. Interleaving of various technologies also increases overhead.
- The memory used depends on the data structures used to maintain template flowsets. Because this implementation does not access the NetFlow cache entry size directly, the memory used is not significant.
- Version 9 slightly decreases overall performance, because generating and maintaining valid template flowsets requires additional processing.
NetFlow Sample Configuration
R4# conf term
R4(config)# int fa 0/0
R4(config-if)# ip flow ingress
R4(config)# int fa 0/1
R4(config-if)# ip flow ingress
R4(config)# ip flow-export source lo 0
R4(config)# ip flow-export version 5
R4(config)# ip flow-export destination 192.168.1.50 5000 ! "5000" is the UDP port to which NetFlow packets are sent (default is 2055)
R4(config)# ip flow-export destination 192.168.2.10 1111 ! it is possible to export netflow to multiple destinations
NetFlow - Flow Cache Check:
R4# show ip cache flowshow ip flow export
IP packet size distribution (171856M total packets):
1-32 64 96 128 160 192 224 256 288 320 352 384 416 448 480
.000 .700 .061 .067 .004 .000 .001 .002 .000 .008 .005 .000 .035 .000 .071
512 544 576 1024 1536 2048 2560 3072 3584 4096 4608
.000 .000 .000 .000 .039 .000 .000 .000 .000 .000 .000
...OUTPUT OMITTED...
Protocol Total Flows Packets Bytes Packets Active(Sec) Idle(Sec)
-------- Flows /Sec /Flow /Pkt /Sec /Flow /Flow
TCP-Telnet 82118 0.0 2 57 0.0 6.8 20.7
TCP-FTP 63484 0.0 1 44 0.0 1.9 21.5
TCP-FTPD 28830 0.0 1 43 0.0 0.1 14.8
TCP-WWW 9164285 2.1 144 199 309.3 5.8 21.4
TCP-SMTP 25278350647 5885.5 4 52 27322.7 3.3 21.7
TCP-X 716507 0.1 1 40 0.1 2.0 23.4
To display the status and the statistics for NetFlow accounting data export, including the main cache and all other enabled caches, use the show ip flow export command in user EXEC or privileged EXEC mode.
Router# show ip flow export0 flows failed due to lack of export packet
Flow export v5 is enabled for main cache
Exporting flows to 10.51.12.4 (9991) 10.1.97.50 (9111)
Exporting using source IP address 10.1.97.17
Version 5 flow records
11 flows exported in 8 udp datagrams
0 flows failed due to lack of export packet
0 export packets were sent up to process level
0 export packets were dropped due to no fib
0 export packets were dropped due to adjacency issues
0 export packets were dropped due to fragmentation failures
0 export packets were dropped due to encapsulation fixup failures
0 export packets were dropped enqueuing for the RP
0 export packets were dropped due to IPC rate limiting
0 export packets were dropped due to output drops
The total number of export packets that were not sent because no memory was available to create an export packet.
0 export packets were sent up to process level
The total number of export packets that could not be processed by CEF or by fast switching, possibly because another feature requires running on the packet.
0 export packets were dropped due to no fib
0 export packets were dropped due to adjacency issues
The total number of export packets that CEF was unable to switch or forward up to the process level.
0 export packets were dropped due to fragmentation failures
0 export packets were dropped due to encapsulation fixup failures
The total number of export packets that were dropped because of problems constructing the IP packet.
0 export packets were dropped enqueuing for the RP
0 export packets were dropped due to IPC rate limiting
The total number of export packets that were dropped because there was a problem transferring the export packet between the RP and the line card.
0 export packets were dropped due to output drops
The total number of export packets that were dropped because the send queue was full while the packet was being transmitted.
Displays the status and statistics for all of the flow exporters configured on a router:
Router# show flow exporterSampling
Flow Exporter FLOW-MONITOR-1:
Description: Exports to the datacenter
Export protocol: NetFlow Version 9
Transport Configuration:
Destination IP address: 172.16.10.2
Source IP address: 172.16.6.2
Source Interface: Ethernet0/0
Transport Protocol: UDP
Destination Port: 650
Source Port: 55864
DSCP: 0x3F
TTL: 15
Output Features: Used
mode (Flexible NetFlow)
To specify the type of sampling and the packet interval for a Flexible NetFlow sampler
mode { deterministic | random } 1 out-of window-size
deterministic - packets are chosen periodically based on the configured interval.
random - packets are chosen in a manner that should eliminate any bias from traffic patterns and counter any attempt by users to avoid monitoring.
1 out-of window-size - Specifies the window size from which to select packets. Range: 2 to 32768
Router(config)# ip cef
Router(config)# flow-sampler-map my-map
Router(config-sampler)# mode random one-out-of 100
If NetFlow is not behaving as expected, consider the following list of common NetFlow troubleshooting targets that could be investigated:
■ No network connectivity exists between a NetFlow router and its configured NetFlow collector
■ The router’s NetFlow configuration is incorrect
■ The NetFlow collector’s configuration is incorrect
■ An ACL or a firewall is blocking NetFlow traffic
IP SLAs
- Cisco IOS IP SLA feature - can be used to measure how the network treats traffic for specific applications
- IP SLA accomplishes this by synthetically generating traffic bearing similar characteristics to application traffic
- This traffic, called probes, is sent to a destination router. This destination router is configured to respond to the received probes with time-stamp information, which can then be used to calculate performance metrics for the traffic.
- IP SLAs can be used for baselining network application performance.
IP SLA Source Configuration
ip sla monitor 1IP SLA Responder Configuration
type tcpConnect dest-ipaddr 10.4.4.4 dest-port 80 source-port 17406
tos 64
ip sla monitor schedule 1 life forever start-time now
!
ip sla monitor responderViewing Information Collected on the IP SLA Source
ip sla monitor responder type tcpConnect ipaddress 10.4.4.4 port 80
!
R1# show ip sla monitor statistics
Round trip time (RTT) Index 1
Latest RTT: 168 ms
Latest operation start time: *16:10:52.453 UTC Sun Mar 3 2002
Latest operation return code: OK
Number of successes: 13
Number of failures: 1
Operation time to live: Forever
BB2# show ip sla monitor responderCommon IP SLA issue
IP SLA Monitor Responder is: Enabled
Number of control message received: 15 Number of errors: 1
Recent sources:
192.168.0.11 [00:38:01.807 UTC Fri Mar 1 2002]
192.168.0.11 [00:37:01.783 UTC Fri Mar 1 2002]
Recent error sources:
192.168.0.11 [00:24:01.807 UTC Fri Mar 1 2002] RTT_FAIL
tcpConnect Responder:
IP Address Port
10.4.4.4
- The source or destination IP addresses configured on the IP SLA source or responder is incorrect.
- The frequency of the probes is set for a value that is too long.
- The schedule is set to begin sending probes at some time in the future.
- The probes are being filtered by an ACL or a firewall.
NBAR - Network-Based Application Recognition
- NBAR can classify various traffic types by examining information at Layers 3 through 7.
- Protocols that change port numbers (that is, stateful protocols) can also be tracked.
- Although Cisco IOS comes with multiple NBAR application signatures, there is a continuing need for additional signature recognition capabilities.
- NBAR signature recognition capability can be expanded by adding one or more PDLM files to a router’s flash.
(Example: you can install a Bit Torrent Packet Description Language Module (PDLM) into the router’s flash)
- NBAR can function as a protocol discovery tool. Therefore, like NetFlow and IP SLA, NBAR can serve as a useful baselining tool.
- The protocol discovery feature of NBAR can be enabled on an interface to determine the applications consuming the most bandwidth on that interface (that is, the top talkers).
NBAR Configuration
Router(config-if)# ip nbar protocol-discovery <-- enable NBAR protocol discoveryNBAR Verification
Router(config)# ip nbar pdlm <pdlm_file> <-- expanding NBAR Recognition by adding one or more PDLM files to a router’s flash.
Router(config)# ip nbar port-map protocol {tcp udp} <port_number> [<port_number>] <-- modify the ports the NBAR uses
R1#sh ip nbar protocol-discovery <-- NBAR collected traffic statistics for an interface
FastEthernet0/0
Last clearing of "show ip nbar protocol-discovery" counters 00:04:38
Input Output
----- ------
Protocol Packet Count Packet Count
Byte Count Byte Count
5min Bit Rate (bps) 5min Bit Rate (bps)
5min Max Bit Rate (bps) 5min Max Bit Rate (bps)
------------------------ ------------------------ ------------------------
icmp 2030 1015
231420 115710
0 0
18000 9000
bgp 0 0
0 0
....
Common NBAR troubleshooting issues:
■ NBAR is not correctly recognizing applications
- This can occur if an application is using a nonstandard port.
- use the show ip nbar port-map protocol command to see what port(s) is associated with a specified application.
- For example, perhaps a web server is using TCP port 8080 instead of port 80.
- issue the show ip nbar port-map http command and see that NBAR is only recognizing TCP port 80 as HTTP traffic.
- use the ip nbar port-map http tcp 80 8080 command in global configuration mode to cause NBAR to recognize either TCP port 80 or 8080 as HTTP traffic.
■ NBAR does not support a specific application
- If a PDLM file exists for an application, you can download it from Cisco.com, copy it to the flash of a router, and reference it with the
ip nbar pdlm <pdlm-file> command in global configuration mode.
■ NBAR degrades router performance
- Depending on the underlying router platform, the performance of a router might suffer as a result of NBAR’s inspection of multiple flows.
- You can use the show processes cpu command to determine the CPU utilization of a router.
QoS
- Modular QoS CLI (MQC)
- AutoQoS Enterprise
AutoQoS Enterprise is configured via a three-step process:
R4(config)# int fa0/0
R4(config-if)# auto discovery qos <--- AutoQoS Enterprise discovery phase
R4# show auto discovery qos <--- After the discovery phase runs for a period of time (at least two or three days)
FastEthernet0/0
AutoQoS Discovery enabled for applications
Discovery up time: 1 minutes, 7 seconds
AutoQoS Class information:
Class Voice:
Recommended Minimum Bandwidth: 5 Kbps/<1% (PeakRate)
Detected applications and data:
Application/ AverageRate PeakRate Total
Protocol (kbps/%) (kbps/%) (bytes)
----------- ----------- -------- ------------
rtp audio 1/<1 5/<1 10138
...
Suggested AutoQoS Policy for the current uptime:
!
class-map match-any AutoQoS-Voice-Fa0/0
match protocol rtp audio
!
policy-map AutoQoS-Policy-Fa0/0
class AutoQoS-Voice-Fa0/0
priority percent 1
set dscp ef
class class-default
fair-queue
R4(config)# int fa 0/0Common reasons that AutoQoS (both AutoQoS VoIP and AutoQoS Enterprise) might not function correctly on a router:
R4(config-if)# auto qos <-- Apply the recommended policy using the auto qos
■ Cisco Express Forwarding (CEF) is not enabled
- AutoQoS can use NBAR to recognize traffic types, and NBAR requires CEF to be enabled. You can enable CEF with the ip cef
■ An interface’s bandwidth is not correctly configured
- Cisco IOS assumes the bandwidth of a serial interface to be 1.544 Mbps (that is, the bandwidth of a T1 circuit).
- Because serial interfaces often run at different speeds, you should configure interfaces bandwidth <bandwidth-in-kbps>
- Some routing protocols (for example, EIGRP and OSPF) can reference this bandwidth amount when calculating a route metric.
- AutoQoS can reference this bandwidth amount to determine which QoS mechanisms should be enabled on an interface.
- If the bandwidth value of an interface is left at its default setting, AutoQoS might not optimally configure QoS on an interface.
■ An interface has not been configured with an IP address
- One QoS mechanism that AutoQoS might configure is Multilink PPP (MLP), which is a link fragmentation and interleaving mechanism.
- Part of an MLP configuration involves the creation of a virtual multilink interface that needs to have an IP address assigned.
- AutoQoS takes the needed IP address from the physical interface being configured for AutoQoS.
- An interface should be configured with an IP address prior to configuring the interface for AutoQoS.
■ Only one side of a link has been configured
- AutoQoS is enabled on an interface.
- However, the interface in the router at the other end of the link needs a complementary configuration.
- For example, consider two routers interconnected via a serial link running at a link speed of 512 kbps. If you configured AutoQoS for the interface at one end of the link, that interface might be configured for QoS mechanisms that include MLP and RTP header compression (cRTP). However, these mechanisms will not function correctly until the interface at the other end of the link is similarly configured.
■ The configuration created by AutoQoS has been modified
- AutoQoS configurations are based on QoS mechanisms available in Cisco IOS.
- Therefore, a configuration generated by AutoQoS can be customized.
- As a result, the underlying issue you are troubleshooting might be caused by the customization of AutoQoS’ configuration
Wireless Troubleshooting Targets
- CCNP R/S focuses on troubleshooting targets on a wired network that could impact a wireless network.
Introducing the Cisco Unified Wireless Network
- Wireless local-area networks (WLAN) offer network access via radio waves.
- Wireless clients (such as PCs or PDAs) access a WAP using half-duplex communication.
- The WAP allows a wireless client to communicate with the wired portion of a network.
Components comprise the Cisco Unified Wireless Network architecture:
■ Wireless clients - A wireless client device is typically an end-user device (such as a PC) that accesses a wireless network.
■ WAP - WAPs offer network access for wireless clients.
■ Wireless network unification - wireless network needs to be integrated (that is, unified) with a wired LAN.
■ Wireless network management - a wireless LAN can use network management solutions to enhance security and reliability and offer assistance in WLAN deployments (ex: Cisco Wireless Control System (WCS).
■ Wireless Mobility - include security threat detection, voice services, location services, and guest access.
- Traditional WLANs use an access point in autonomous mode, where the access point is configured with a service set identifier (SSID), radio frequency (RF) channel, and RF power settings.
- Aside from autonomous mode, Cisco Unified Wireless Networks can alternatively operate in split-MAC mode (WAP is considered a LWAP - lightweight access point - which cannot function without a WLC).
- WLAN client sending traffic to the wired LAN sends a packet to a lightweight access point, which encapsulates the packet using Lightweight Access Point Protocol (LWAPP).
- The encapsulated traffic is sent over an LWAPP tunnel to a WLC.
- LWAPP sends packets in a Layer 2 frame with an Ethertype of 0xBBBB.
- LWAPP data traffic uses a UDP destination port of 12222, whereas LWAPP control traffic uses a UDP destination port of 12223.
A lightweight access point - performs functions such as beaconing, packet transmission, and frame queuing.
WLC - assumes roles such as authentication, key management, and resource reservation.
- WLC uses an Extensible Authentication Protocol (EAP) to communicate with the authentication server.
- Cisco Secure Access Control Server (ACS) can, for example, act as an authentication server.
Supported EAP types include the following:
■ EAP-Transport Layer Security (EAP-TLS)
- Wireless clients and authentication servers mutually authenticate using digital certificates.
■ EAP-Protected EAP (EAP-PEAP)
- The authentication server (that is, a RADIUS server) is authenticated over a Transport Layer Security (TLS) tunnel using a digital certificate,
- Wireless clients are authenticated via Extensible Authentication Protocol—Generic Token Card (EAP-GTC) or Extensible Authentication Protocol—
Microsoft Challenge Handshake Authentication Protocol version 2 (EAP-MSCHAPv2).
■ EAP Tunneled Transport Layer Security (EAP-TTLS)
- The RADIUS server is authenticated over a TLS tunnel using the certificate of the server, and wireless clients authenticate using username and password.
■ Cisco Lightweight Extensible Authentication Protocol (LEAP)
- Cisco developed LEAP as an early and proprietary EAP method; however, LEAP’s vulnerability to a dictionary attack represents a major LEAP weakness.
■ Cisco EAP-Flexible Authentication via Secure Tunneling (EAP-FAST)
- Cisco proposed EAP-FAST to address weaknesses of LEAP.
WLAN controller components:
■ Ports - A port on a WLAN controller physically connects the WLAN controller to the wired network (for example, to a Cisco Catalyst switch port).
■ Interfaces:An interface of a WLAN controller logically maps to a VLAN on a wired network.
■ WLANs:A WLAN can be configured with security features, QoS mechanisms, and other wireless network parameters. Also, a WLAN associates an SSID to a
WLC interface.
Wired Network Issues Impacting Wireless Networks
PoE
- WAPs require power; however, these access points might need to be installed away from power outlets.
- If the Cisco Catalyst switch is not appropriately configured to provide PoE, or if the switch has no additional power available, an attached WAP might fail to power on.
- PoE troubleshooting aid is the Cisco Power Calculator, an online tool available at http://tools.cisco.com/cpc/launch.jsp (can help determine the power capacity of a switch).
VLANs
- wireless users might experience connectivity issues if traffic from their wireless VLAN is not permitted over a trunk in the wired network.
- trunk configurations on Cisco Catalyst switches might need to be inspected as part of troubleshooting wireless connectivity issues.
show vlanSecurity
show interfaces trunk
show interfaces switchport
- ACLs might inadvertently be configured to block traffic that should be permitted on a network.
- In the case of wireless networks configured in a splitMAC architecture, recall that UDP ports 12222 and 12223 should be permitted between a WAP and a WLC.
DHCP
- clients might need to roam from one subnet to another. In such an instance, a loss of wireless connectivity might result from a DHCP issue.
DHCP Troubleshooting Commands
show ip dhcp conflict Lists any IP address conflicts identified by a router (via ping or gratuitous ARP)QoS
show ip dhcp binding Displays IP addresses assigned by an IOS DHCP server, MACs and lease expirations
clear ip dhcp binding * Releases all current DHCP leases
clear ip dhcp conflict * Clears all currently identified DHCP conflicts
debug ip dhcp server events Provides real-time information about DHCP address assignments and database updates
debug ip dhcp server packet Displays real-time decodes of DHCP packets
- Latency-sensitive traffic traveling over a wireless network (for example, Voice over Wireless LAN [VoWLAN]) might suffer from poor performance if QoS markings are not preserved as traffic crosses the boundary between the wireless and wired portions of a network.
- mls qos trust dscp - interface configuration mode command you can issue this command on a Cisco Catalyst switch to cause an interface to trust incoming DSCP markings and preserve priority markings on wireless traffic as it enters a wired network, you can issue this command on a Cisco Catalyst switch port that connects to a WLC.