CCNP Tshoot - Large Enterprise Network Troubleshooting


WAN Connections
 - Most large enterprise networks connect to one or more remote office locations, typically accessible via an IP WAN.
 - Consider quality of service (QoS) and security issues.
 - The relatively limited bandwidth available on an IP WAN might create quality issues for latency-sensitive applications such as streaming voice or video.
 - Security mechanisms should be in place to secure the traffic flowing between the headquarters and remote locations (site-to-site VPNs and remote-access VPN).

Remote Office Troubleshooting
 - Large enterprise networks often contain multiple remote offices connecting back to a network located at a corporate headquarters.
 - Troubleshooting remote office (Branch) network issues can require knowledge of a wide array of technologies:
 - Services (for Example, IP Telephony)
 - Security (for Example, ACLs)
 - LAN Services (for Example, DHCP, VRRP, NAT)
 - IP Routing (for Example, Routing over a VPN)
 - IP WAN (for Example, VPNs)

VPN Types
■ Site-to-site VPNs
 - typically terminates in a router at the head quarters and a router at the remote site.
 - such an arrangement does not require the clients at the remote site to have VPN client software installed.
■ Remote-access VPNs
 - requires VPN clients at the remote site to run VPN client software.
 - Although this approach might require more administrative overhead to install client software on all clients, remote-access VPNs do offer more flexibility for mobile users.
 - For example, clients can connect via their hotel’s Internet connection using VPN client software on their laptop computer

Site-to-Site VPN Considerations (listing of potential issues):
■ Overlapping IP address spaces
  - Branch A and Headquarters locations could have an overlapping IP address space (that is, 10.1.1.0/24).
 - This overlap might prevent these two networks from communicating successfully.
 - A fix for such an issue is to configure Network Address Translation (NAT) to support overlapping networks.
■ Dynamic routing protocols
 - EIGRP, OSPF, and RIPv2 typically send advertisements to a multicast address; however, IPsec tunnels transport only unicast IP packets.
 - A Generic Routing Encapsulation (GRE) tunnel, however, can transport a variety of traffic types.
 - Therefore, all IP traffic (including multicast and broadcast traffic) can initially be encapsulated within GRE packets, which are unicast IP packets.
 - Those GRE packets can then be encapsulated inside IPsec packets to secure their transmission.
■ Maximum transmission unit (MTU) size
 - Most Cisco router interfaces default to an MTU size of 1500 bytes for packets (that is, not including a Layer 2 header).
 - However, when traffic is encapsulated inside a VPN tunnel, the tunnel header(s) add to the packet size.(GRE+IPsec can add 60-80 bytes of overhead to a packet)
 - As a result, the packet size might exceed its MTU setting. When an interface attempts to transmit a packet that exceeds the MTU of the interface, the interface attempts to fragment the packet. If successful, each fragment receives its own header creating a new packet, which is of an acceptable size.
 - However, fragmenting large packets can cause issues. First, the act of performing fragmentation increases the burden on a router processor.
 - Additionally, some packets are marked with a Do Not Fragment (DF) bit, which can cause those packets to be dropped.
■ Misconfiguration
 - The configuration of IPsec tunnels can be quite complex. As a result, a common troubleshooting issue for site-to-site VPNs is a misconfiguration of the VPN endpoints
■ Point-to-point nature of GRE tunnels
 - Because GRE tunnels are point-to-point logical connections, suboptimal pathing might result (excessive delay and poor performance might result).
 - Another option is to create a full mesh of VPN connections (do not scale well).
 - Rather than creating a full mesh of VPN connections between all sites in an enterprise network, you can alternatively use Dynamic Multipoint VPN(DMVPN) technology. DMVPM allows VPN connections to be dynamically created on an as-needed basis.
 - DMVPN solution overcomes the performance issues of a hub-and-spoke topology, while simultaneously overcoming the scalability issues presented by a full mesh topology.
 ■ Suboptimal routing
 - tunnel is a logical connection between two end points; however, that logical connection can span multiple router hops.
 - If a portion of a tunnel spans a slow or unreliable link, the result can be poor performance for all tunnel traffic.
 - Another issue that can lead to suboptimal routing is recursive routing (the tunnel interface might experience flapping. Therefore, poor VPN performance can be linked to an inappropriate routing configuration on one or both of the VPN routers).
■ Route processor overhead
 - Depending on the security algorithms chosen to protect an IPsec tunnel, some router platforms might suffer from poor performance.
 - The number of VPN tunnels that can be terminated on a router depends on the underlying router platform.
Router       Maximum IPsec Speed      Number of Supported VPN Tunnels
Cisco 1841   95 Mbps IPsec VPN        800 tunnels
Cisco 2801   100 Mbps IPsec VPN       1500 tunnels
Cisco 2811   30 Mbps                  1500 tunnels
Cisco 2821   140 Mbps                 1500 tunnels
Cisco 2851   145 Mbps                 1500 tunnels
Cisco 3825   175 Mbps                 2000 tunnels
Cisco 3845   185 Mbps                 2500 tunnels
Remote-Access VPN Considerations
■ Authentication:
 - Users connecting from their PC (running VPN client software) require user credentials (for example, username and password credentials) to gain access to a network.
 - the users might provide correct credentials, but the authentication server might be configured incorrectly or might not be functioning.
■ User profiles
 - Because users log into a remote-access VPN, different users can be assigned different policies through the use of user profiles. As a result, when remote access VPN users are unable to connect to desired resources, the underlying issue might be their user profile.
■ MTU size
 - Remote-access VPN clients have a similar issue with MTU sizes and fragmentation, as previously described for site-to-site VPNs.
 - Fortunately, VPN client software often allows you to configure the MTU size of a tunnel.
■ Misconfiguration VPN software running on a client machine
 - often has multiple configuration options. As a result, a common issue for remote-access VPNs is the misconfiguration of the VPN client software.
■ Client security software
 - Security software running on a client machine might deny traffic required for VPN establishment.
 - Therefore, firewall and anti-virus software running on a VPN client machine might result in the failure of a VPN connection.

Troubleshooting VPN Issues
■ How is IP addressing assigned? (For example, do overlapping IP address ranges exist?)
■ Is the VPN site-to-site or remote-access?
■ How are the MTU values configured on the router interfaces transited by the VPN?
■ What translations (if any) is NAT performing?
■ Are routing protocols routing traffic over a GRE tunnel or over a physical interface?
■ According to a router’s IP routing table, is the best path to a tunnel destination’s IP address the tunnel interface? (If so, a recursive routing issue might result.)
show crypto ipsec sa
!Displays IPsec security association settings

show crypto engine connections active
! Displays configuration information for all active IPsec sessions

show crypto map
! Displays the crypto map configuration of a router (for example, information about ACLs being referenced by the crypto map,
!  the IP address of the IPsec peer, the security association lifetime, and the name of the crypto map transform set)

show ip route
! Displays routes injected into a router’s IP routing table, including next-hop IP address or exit interface information for IP routes

show ip protocols
! Displays information about the active IP routing processes of a router

show interfaces tunnel <number>
! Displays status and configuration information for a specified tunnel interface on a router
Example config:
HeadQuarter
HQ# show run
...OUTPUT OMITTED...
hostname HQ
!
crypto isakmp policy 1
encr 3des
authentication pre-share
group 2
!
crypto isakmp policy 2
encr aes
authentication pre-share
group 2
crypto isakmp key cisco address 172.16.1.2
!
crypto ipsec transform-set TSHOOT-TRANSFORM esp-aes esp-sha-hmac
!
crypto map SDM_CMAP_1 1 ipsec-isakmp
description Tunnel to172.16.1.2
set peer 172.16.1.2
set transform-set TSHOOT-TRANSFORM
match address 100
!
interface Tunnel0
ip address 10.1.1.1 255.255.255.252
ip mtu 1420
tunnel source 172.16.1.1
tunnel destination 172.16.1.2
tunnel path-mtu-discovery
crypto map SDM_CMAP_1
!
interface FastEthernet0/0
ip address 192.168.1.29 255.255.255.0
!
interface Serial1/0
ip address 172.16.1.1 255.255.255.0
encapsulation ppp
crypto map SDM_CMAP_1
!
ip route 0.0.0.0 0.0.0.0 Tunnel0
ip route 10.1.1.0 255.255.255.0 172.16.1.2
ip route 172.16.1.2 255.255.255.255 Serial1/0
!
!
access-list 100 remark SDM_ACL Category=4
access-list 100 permit gre host 172.16.1.1 host 172.16.1.2
!
...OUTPUT OMITTED...
BRANCH
BR# show run
...OUTPUT OMITTED...
hostname BR
!
crypto isakmp policy 1
encr 3des
authentication pre-share
group 2
!
crypto isakmp policy 2
encr aes
authentication pre-share
group 2
crypto isakmp key cisco address 172.16.1.1
!
crypto ipsec transform-set TSHOOT-TRANSFORM esp-aes esp-sha-hmac
!
crypto map SDM_CMAP_1 1 ipsec-isakmp
set peer 172.16.1.1
set transform-set TSHOOT-TRANSFORM
match address SDM_1
!
interface Tunnel0
ip address 10.1.1.2 255.255.255.252
ip mtu 1420
tunnel source 172.16.1.2
tunnel destination 172.16.1.1
tunnel path-mtu-discovery
crypto map SDM_CMAP_1
!
interface FastEthernet0/0
ip address 10.2.2.1 255.255.255.0
!
interface Serial1/0
ip address 172.16.1.2 255.255.255.0
encapsulation ppp
!
ip route 0.0.0.0 0.0.0.0 Tunnel0
ip route 10.1.1.0 255.255.255.0 172.16.1.1
ip route 172.16.1.1 255.255.255.255 Serial1/0
!
ip access-list extended SDM_1
remark SDM_ACL Category=4
permit gre host 172.16.1.2 host 172.16.1.1
...OUTPUT OMITTED...
Verify
Crypto IPSec
HQ# show crypto ipsec sa
interface: Serial1/0
    Crypto map tag: SDM_CMAP_1, local addr 172.16.1.1
...
local crypto endpt.: 172.16.1.1, remote crypto endpt.: 172.16.1.2
..
   transform: esp-aes esp-sha-hmac ,
    in use settings ={Tunnel, }
    conn id: 2002, flow_id: SW:2, crypto map: SDM_CMAP_1
    sa timing: remaining key lifetime (k/sec): (4451479/3185)
    IV size: 16 bytes
    replay detection support: Y
    Status: ACTIVE 

 outbound esp sas:
   transform: esp-aes esp-sha-hmac ,
    ...
    Status: ACTIVE

interface: Tunnel0
  Crypto map tag: SDM_CMAP_1, local addr 172.16.1.1
  protected vrf: (none)
  local ident (addr/mask/prot/port): (172.16.1.1/255.255.255.255/47/0)
  remote ident (addr/mask/prot/port): (172.16.1.2/255.255.255.255/47/0)
  current_peer 172.16.1.2 port 500 
 Crypto Engine
HQ# show crypto engine connections active
ID Interface IP-Address State Algorithm Encrypt Decrypt
1 Serial1/0 172.16.1.1 set HMAC_SHA+3DES_56_C  0 0
2001 Serial1/0 172.16.1.1 set AES+SHA  28 0
2002 Serial1/0 172.16.1.1 set AES+SHA 0 0
 Crypto Map
HQ# show crypto map
Crypto Map “SDM_CMAP_1” 1 ipsec-isakmp
Description: Tunnel to172.16.1.2
Peer = 172.16.1.2
Extended IP access list 100
    access-list 100 permit gre host 172.16.1.1 host 172.16.1.2
Current peer: 172.16.1.2
Security association lifetime: 4608000 kilobytes/3600 seconds
PFS (Y/N): N
Transform sets={
    TSHOOT-TRANSFORM,
}
Interfaces using crypto map SDM_CMAP_1:
    Serial1/0
   Tunnel0 
Tunnel interface
HQ# show interfaces tunnel 0
Tunnel0 is up, line protocol is up
  Hardware is Tunnel
  Internet address is 10.1.1.1/30
  MTU 1514 bytes, BW 9 Kbit, DLY 500000 usec, reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation TUNNEL, loopback not set
  Tunnel source 172.16.1.1, destination 172.16.1.2
  Tunnel protocol/transport GRE/IP 
...

Complex Network Troubleshooting
  - In some cases the troubleshooter might not have the depth of knowledge required in a specific technology. In such a situation, the troubleshooter should know how to obtain the required knowledge or identify who can provide the required expertise.
 - Complex enterprise networks are composed of multiple technologies that might reside at various layers of the OSI model. Knowing which technologies correspond to which OSI layers can help a troubleshooter better understand how issues with one technology can impact other technologies.

OSI Layers of Various Networking Technologies