- Unlike OSPF and EIGRP, BGP does not consider a link’s bandwidth when making a routing decision.
Layer 3 Troubleshooting
When troubleshooting connectivity issues for an IP-based network, the network layer (that is, Layer 3) of the OSI reference model is often an appropriate place to begin your troubleshooting efforts.
Router have two data structures:
■ RIB (IP routing table) - maintained by a router’s control plane
When a router needed to route an IP packet, it consulted its IP routing table to find the best match:
a) route that has the longest prefix,
b) if 2 or more routes have same prefix length, choose route known via protocol with lowest AD (Administrative distance),
c) if 2 or more routes have same prefix length, same AD, choose route with lowest metric,
d) if 2 or more routes have same prefix length, same AD, same metric, then load balance between routes.
■ Layer 3 to Layer 2 mapping
- For multiacces network: ARP cache is the Layer 3 to Layer 2 mapping data structure used for Ethernet-based networks, similar data structures are used for Frame Relay and ATM point-to-multipoint links.
- For point-to-point links: an egress interface might be shown in the IP routing table, as opposed to a next-hop IP address. For these types of links (for example, point-to-point Frame Relay or ATM PVC, HDLC, or PPP links), the information required to construct an outgoing frame can be gleaned from the egress interface, thus not requiring a nexthop IP address.
The lower the administrative distance, the more preferred the route.
0 Connected interfaceCEF (operates in the data plane)
1 Static route (via next-hop or exit-interface)
5 Enhanced Interior Gateway Routing Protocol (EIGRP) summary route
20 External Border Gateway Protocol (BGP)
90 Internal EIGRP
100 IGRP
110 OSPF
115 Intermediate System-to-Intermediate System (IS-IS)
120 Routing Information Protocol (RIP)
140 Exterior Gateway Protocol (EGP)
160 On Demand Routing (ODR)
170 External EIGRP
200 Internal BGP
255 Unknown* (Unreachable)
- Continually querying a router’s routing table and its Layer 3 to Layer 2 mapping data structure is less than efficient.
- Cisco Express Forwarding (CEF) makes lookups much more efficient.
- CEF gleans its information from the router’s IP routing table and Layer 3 to Layer 2 mapping tables.
- Then, CEF’s data structures can be referenced when forwarding packets.
CEF have two data structures:
- Forwarding Information Base (FIB):The FIB contains Layer 3 information, similar to the information found in an RIB. Additionally, a FIB contains information about multicast routes and directly connected hosts.
- Adjacency table:When a router is performing a route lookup using CEF, the FIB references an entry in the adjacency table. The adjacency table entry contains the
frame header information required by the router to properly form a frame. Therefore, an egress interface and a next-hop IP address would be in an adjacency entry for a
multipoint interface, whereas a point-to-point interface would require only egress interface information.
Troubleshooting Layer 3 Forwarding Information
show ip route <ip-address>Troubleshooting Layer 3 to Layer 2 Mapping Information
! Displays a router’s best route to the specified IP address
show ip route <network> <subnet-mask>
! Displays a router’s best route to the specified network, if the specific route (with a matching subnet mask length) is found in the router’s IP routing table.
show ip route <network> <subnet-mask> longer-prefixes
! Displays all routes in a router that are encompassed by the specified network(useful when troubleshooting route summarization issues.
show ip cef <ip-address>
! Displays information (for example, next-hop IP address and egress interface) required to forward a packet
show ip cef <network> <subnet-mask>
! Displays information from a router’s FIB showing the information needed to route a packet to the specified network with the specified subnet mask.
show ip cef exact-route <source-ip-address> <destination-ipaddress>
! Displays the adjacency that will be used to forward a packet from the specified source IP address to the specified destination IP address.
! useful if the router is load balancing across multiple adjacencies, and you want to see which adjacency will be used for a certain combination of src/dst IP addr
show ip arpAsymmetric Routing and (Unknown) Unicast Flooding
! Displays a router’s ARP cache, containing IP address to MAC address mappings.
clear ip arp
! Clear ARP cache and allow a router to relearn information after you make a topology change
show frame-relay map
! Displays Frame Relay DLCIs associated with different next-hop IP addresses.
show adjacency detail
! Displays the frame headers in a router’s CEF adjacency table used to encapsulate a frame being sent to an adjacency
https://www.cisco.com/c/en/us/td/docs/solutions/Enterprise/Campus/HA_campus_DG/hacampusdg.html#wp1108782
Asymmetric routing: a packet traverses from a source to a destination in one path and takes a different path when it returns to the source.
This is commonly seen in Layer-3 routed networks.
- is not a problem by itself, but will cause problems when Network Address Translation (NAT) or firewalls are used in the routed path.
- is not a problem for current TCP/IP implementations, since TCP connection does not care which route a certain IP packet takes as long as it reaches its destination in a reasonable time.
- in general is a normal, but unwanted situation in an IP network.
- is very common with BGP, and completely avoiding it is impossible.
- Traceroute is the best way to make sure that your network traffic is traversing the expected path.
Issues when using VRRP/HSRP/GLBP:
- Avoid asymmetric routing and unicast flooding; do not span VLANs across the access layer.
When a less-than-optimal topology is used, a long-existing but frequently misunderstood situation can occur as a result of the difference between ARP and CAM table aging timers. If VLANs span across multiple access layer switches, return path traffic can be flooded to all access layer switches and end points. This can be easily avoided by not spanning VLANs across access layer switches. If this cannot be avoided, then tune the ARP aging timer so that it is less than the CAM aging timer.
Workaroutd:
A shorter ARP cache timer causes the standby HSRP peer to ARP for the target IP address before the CAM entry timer expires and the MAC entry is removed. The subsequent ARP response repopulates the CAM table before the CAM entry is aged out and removed. This removes the possibility of flooding asymmetrically-routed return path traffic to all ports.
Set ARP timeout <= MAC (CAM) timeout
Default values:
ARP timeout - 14400 sec (4 hours)
MAC timeout - 300 sec (5 min)
Cisco Debug Condition Interface
- allows you to only show the debug output from a specific interface.
- other conditions you can filter on as well including application, mac-address, vlan and there are a number of voice related commands.
R#debug condition interface fa0/1
Condition 1 set
!
R#debug ip ospf hello
OSPF hello events debugging is on
!
R#
*Mar 1 00:59:01.573: OSPF: Rcv hello from 150.6.4.4 area 45 from FastEthernet0/1 183.6.45.4
*Mar 1 00:59:01.573: OSPF: End of hello processing
http://www.rogerperkin.co.uk/ccie/debug-condition-interface-cisco/
EIGRP Troubleshooting
- EIGRP advertises routes to directly attached neighbors, like a distance vector routing protocol, while using a series of tables, similar to a link-state routing protocols,
- EIGRP also offers the benefit of fast convergence after a link failure.
- Load balancing is supported over both equal-cost paths (a default behavior) and unequal-cost paths (through the variance feature).
- adjacency - need to establish a relationship with a neighboring router before exchanging route information with that neighbor.
- To turn off automatic summarization, you can issue the no auto-summary
Data Structures of EIGRP
1) EIGRP interface table
- All of a router’s interfaces that have been configured to participate in an EIGRP routing process are listed in this table. (excluding passive-interface)
2) EIGRP neighbor table
- lists a router’s EIGRP neighbors.
- a neighbor is removed from this table if the neighbor has not been heard from for a period of time defined as the hold-time.
- if an interface, from which a neighbor is known, is removed from the EIGRP interface table because it goes down, the neighbor is removed from this table unless there is a multiple link and one of the interfaces is still up. In that case, the second interface will still provide the neighborship.
3) EIGRP topology table
- contains routes learned by a router’s EIGRP routing process.
- the best route for a network in this table becomes a candidate to be injected into the router’s IP routing table.
- If multiple routes in this table have an equal metric, or if EIGRP’s variance feature is configured, more than one route might become candidates for injection into the IP routing table, but only to a maximum of 4 by default.
EIGRP Troubleshooting Commands
show ip eigrp interfaces <-- displays all of a router’s interfaces configured to participate in an EIGRP routing process (with the exception of passive interfaces)
show ip eigrp neighbors <---shows a router’s EIGRP neighbors
show ip eigrp topology <---displays routes known to a router’s EIGRP routing process. These routes are contained in the EIGRP topology table
show ip route eigrp <---shows routes known to a router’s RIB that were injected by the router’s EIGRP routing process
debug ip routing <---displays updates that occur in a router’s IP routing table
debug eigrp packets <---used to display all EIGRP packets exchanged with a router’s EIGRP neighbors (can be narrowed to only display specific EIGRP packet)
debug ip eigrp <-- shows information contained in EIGRP packets and reveals how an EIGRP routing process responds to that information
OSPF Troubleshooting
- OSPF is a nonproprietary link-state protocol.
- Like EIGRP, OSPF offers fast convergence and is a popular enterprise routing protocol.
- OSPF uses a metric of cost, which is a function of bandwidth. Cost can be calculated as follows: cost = 100,000,000 / bandwidth (in kbps)
- Adjacencies are not established upon the immediate receipt of Hello messages
OSPF Data Structures
1) OSPF interface table
All the router interfaces that have been configured to participate in an OSPF routing process are listed in this table.
2) OSPF neighbor table
- OSPF neighbors learned via Hello packets.
- A neighbor is removed from this table if Hellos have not been heard from the neighbor within the dead time interval.
- A neighbor is removed from this table if the interface associated with the neighbor goes down.
3) OSPF link-state database
- contains topology information for all areas in which a router participates, in addition to information about how to route traffic to networks residing in other areas or autonomous systems.
- if a router is participating in more than one OSPF area, the router contains more than one OSPF link-state database (one for each area).
- all routers participating in that OSPF area should have identical OSPF linkstate databases.
- Information about redistributed routes is stored in an area separate from the area-specific OSPF linkstate databases.
4) OSPF Routing Information Base
The OSPF Routing Information Base (RIB) stores the results of the OSPF shortest path first (SPF) calculations.
Neighbor states
OSPF Neighbor Problems Explained - www.cisco.com/c/en/us/support/docs/ip/open-shortest-path-first-ospf/13699-29.html
OSPF Troubleshooting Commands
show ip ospf interface [brief] displays all of a router’s interfaces configured to participate in an OSPF routing process
show ip ospf neighbor displays the state of OSPF neighbors learned off a router’s active OSPF interfaces.
show ip ospf statistics provides information about how frequently a router is exe cuting the SFP algorithm and when the SPF algorithm last ran
show ip ospf database displays the LSA headers contained in a router’s OSPF link-state database
show ip route ospf shows routes known to a router’s IP routing table that were learned via OSPF
debug ip ospf monitor provides real-time updates showing when a router’s SPF algorithm is scheduled to run
debug ip routing displays updates that occur in a router’s IP routing table
debug ip ospf packet shows the transmission and reception of OSPF packets in real time (useful for monitoring Hello's)
debug ip ospf adj provides real-time updates about the formation of an OSPF adjacency (OSPF states)
debug ip ospf events shows real-time OSPF events (IN/OUT of Hello's and LSAs)
(useful on a router that appears to be ignoring Hello messages received from a neighboring router)
show ip ospf virtual-links information about OSPF virtual-links that are required for areas not physically adjacent to the backbone area (area 0)
Route Redistribution Troubleshooting
- only routes in a router’s IP routing table can be redistributed
- allows routes learned via one method (for example, statically configured, locally connected, or learned via a routing protocol) to be injected into a different
routing protocol.
- if two routing protocols are mutually redistributed, the routes learned via each routing protocol are injected into the other routing protocol.
- boundary router - a router that sits at the boundary of the routing domains to be redistributed.
- when a route is injected into a routing protocol, a metric used by the destination routing protocol needs to be associated with the route being injected
- seed metric - the metric assigned to a route being injected into another routing process
- seed metric is needed to communicate relative levels of reachability between dissimilar routing protocols.
- seed metric can be defined in one of three ways:
--- default-metric command,
--- metric parameter in the redistribute command,
--- a route map configuration.
- if a seed metric is not specified, a default seed metric is used.
- RIP and EIGRP have a default metric that is considered unreachable. (if you do not configure a nondefault seed metric when redistributing routes into RIP or EIGRP, the redistributed route will not be reachable).
- the capability to distinguish between internal and external routes can help prevent a potential routing loop, where two routing protocols continually redistribute a route into one another.
Route Redistribution Troubleshooting Targets
- Effective troubleshooting of route redistribution requires knowledge of verification and troubleshooting commands for each routing protocol.
- Cisco IOS IP route profiling feature to troubleshoot route instability
- The IP route profiling feature measures the number and type of IP routing table updates every 5 seconds.
- Route profiling can be enabled in global configuration mode with the
R(conf_t)# ip route profile <--- enable Cisco IOS IP route profiling- The left column in the output of the show ip route profile command represents the number of changes that occurred during a 5-second interval.
R4# show ip route profile
IP routing table change statistics:
Frequency of changes in a 5 second sampling interval
-------------------------------------------------------------
Change/ Fwd-path Prefix Nexthop Pathcount Prefix
interval change add change change refresh
-------------------------------------------------------------
0 38 38 41 41 41
1 3 3 0 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
10 0 0 0 0 0
15 0 0 0 0 0
20 0 0 0 0 0
25 0 0 0 0 0
30 0 1 0 1 0
55 0 0 0 0 0
...
10155 0 0 0 0 0
13280 0 0 0 0 0
Overflow 0 0 0 0 0
Consider row in the output that has a 30 in the left column :
- the number 1 under the Prefix Add column indicates that during one 5-second interval, 30–54 prefixes were added to the IP routing table.
- notice that the next value in the Change/Interval column after 30 is 55.
- a number appearing in the 30 row indicates during how many 5-second timing intervals a particular IP routing update occurred 30–54 times.
Ideally, only numbers in the first row (that is, the 0 row) should change in a stable network.
If numbers in other rows change, a routing loop might be occurring.
Route Redistribution with EIGRP and OSPF
Route Redistribution Verification and Troubleshooting Syntax
Insert EIGRP routes to OSPF
router ospf <process-id> <---Global configuration mode command that enables an OSPF process on a routerInsert OSPF routes to EIGRP
redistribute eigrp <ASN> subnets <---Router configuration mode command that redistributes routes, including subnets
default-metric <metric> <---Router configuration mode command that specifies the metric used for EIGRP routes redistributed into OSPF
R2(config-router)#redistribute eigrp 100 metric 64 subnets
router eigrp <ASN> <---enables an EIGRP routing process on a router
redistribute ospf <PID> <---redistributes routes from a specified OSPF process ID into EIGRP
default-metric <BW> <delay> <reliability> <load> <mtu> <---parameters used to calculate the seed metric for routes redistributed into EIGRP
bandwidth(in kbps)
delay(in tens of microseconds)
reliability(maximum of 255)
load(minimum of 1)
mtu(in bytes)
R2(config-router)#default-metric 1500 100 255 1 1500
BGP Troubleshooting
- An IGP is used within an autonomous system (AS), where an autonomous systemis defined as a network under a single administrative control.
- BGP - an exterior gateway protocol (EGP) - is a routing protocol typically used between autonomous systems.
- Unlike OSPF and EIGRP, BGP does not consider a link’s bandwidth when making a routing decision.
- A BGP router always learns its neighbors through manual configuration of those neighbors as opposed to dynamically learning about neighbors.
- A BGP router attempts to establish a session with its configured neighbors using TCP port 179.
- After a session has been established, BGP OPEN messages are exchanged to communicate each neighbor’s BGP characteristics.
BGP Data Structures (how IP routing protocol interact with an IP routing table):
1) Incoming Route Information
- A BGP router receives BGP updates from a BGP neighbor.
- Unlike OSPF and EIGRP neighbors, BGP neighbors do not need to be directly connected.
- Rather, BGP neighbors can be multiple hops away from one another.
- BGP neighbors are often referred to as peers.
2) Data Structure of IP Routing Protocol
- BGP maintains two data structures: the neighbor table and the BGP table.
- Neighbor table contains status information about BGP neighbors,
- BGP table contains network prefixes learned from BGP neighbors.
3) Injecting and Redistributing Routes
- Routes can be inserted in the BGP table by advertise ments received from BGP neighbors or by locally injected routes.
- For a route to be locally injected (either through a manual configuration or through a redistribution configuration), it must be present in the IP routing table.
4) Route Installation
- Similar to OSPF and EIGRP, BGP might have more than one route to a network prefix in its BGP table.
- BGP then selects what it considers to be the best route to that network prefix; that best route becomes a candidate to be inserted into the IP routing table.
5) Outgoing Route Information
- Routes in a router’s BGP table that are considered the best routes to their network prefixes are advertised to the router’s BGP peers.
- BGP offers several features to limit routes advertised to BGP peers or received from BGP peers.
BGP prefers the path:
1. highest weight. Note that the BGP weight parameter is a Cisco-specific parameter.
2. highest local preference value.
3. originated by BGP on the local router.
4. shortest autonomous system.
5. lowest origin type. (NOTE: IGP < EGP < INCOMPLETE.)
6. lowest multi-exit discriminator (MED).
7. prefer eBGP paths over iBGP paths.
8. lowest IGP metric to the BGP next-hop.
9. lowest BGP router ID.
BGP Best Path Selection Algorithm
http://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/13753-25.html
Reasons why the peering of two BGP routers might fail:
■ TCP establishes a BGP session. Therefore, a lack of IP connectivity between two BGP routers prevents a peering relationship from forming between those routers.
■ The AS numbers must match between the AS number in messages received from a neighbor and the AS number a router has configured for that neighbor.
If the AS numbers fail to match, the session is reset.
■ A BGP router might have multiple active IP addresses configured across its various interfaces.
A router might send a BGP message from one of its IP addresses that does not match the IP address configured for that router on its peer. If the peer does not
recognize the source IP address of the BGP message, the peering relationship fails.
- After initially establishing a peering relationship, two BGP peers exchange information in their BGP tables.
- Incremental updates are sent thereafter.
- If a network prefix is removed from the BGP table of a router, that router sends a WITHDRAW message to appropriate peers.
BGP Troubleshooting Commands
show ip bgp summary
! Displays a router’s BGP router ID, AS number, information about the BGP’s memory usage, and summary information about BGP neighbors
show ip bgp neighbors
! Detailed information about all the BGP neighbors of a router
show ip bgp
! Displays the network prefixes present in the BGP table
debug ip routing
! Displays updates that occur in a router’s IP routing table. Therefore, this command is not specific to BGP
show ip route bgp
! Shows routes known to a router’s IP routing table that were learned via BGP
debug ip bgp
! Although this command does not show the contents of BGP updates,
! the output does provide real-time information about BGP events, such as the establishment of a peering relationship.
debug ip bgp updates
! shows real-time information about BGP updates sent and received by a BGP router
BGP Configuration of ASPATH and Local Preference BGP Attributes
route-map <NAME> [permit|deny] [seq-num] <---Creates a route map
set local-preference <local-preference> <---Sets the local preference for routes matched by a route-map
set as-path prepend <ASN-1> [...ASN-n] <---Defines an AS path to prepend to an AS path known by the BGP table
router bgp <as-number> <---Enables a BGP process for a specific autonomous system
neighbor <peer-IP> route-map <NAME> [in| out] <---Apply a route map to routes received from or advertised to a BGP peer
BGP Outbound Influence: Local-pref config example
R2 <-- 64kbps-> BB1
<--128kbps-> BB2
R2#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 172.16.1.1 0 100 0 65002 i
* 172.16.2.2 0 100 0 65003 i
R2(config)#route-map LOCALPREF-BB1
R2(config-route-map)#set local-preference 100
R2(config)#route-map LOCALPREF-BB2
R2(config-route-map)#set local-preference 200
R2(config)#router bgp 65001
R2(config-router)#neighbor 172.16.1.1 route-map LOCALPREF-BB1 in
R2(config-router)#neighbor 172.16.2.2 route-map LOCALPREF-BB2 in
R2#show ip bgp
Network Next Hop Metric LocPrf Weight Path
* 0.0.0.0 172.16.1.1 0 100 0 65002 i
*> 172.16.2.2 0 200 0 65003 i
BGP Inbound Influence: AS-Path prepend config example
R2 <---> BB1
BB1#show ip bgp
Network Next Hop Metric LocPrf Weight Path
* 10.1.1.1/32 10.1.3.2 0 65003 65001 ?
*> 172.16.1.2 11 0 65001 ?
R2(config)#route-map ASPATH 10
R2(config-route-map)#set as-path prepend 65001 65001
R2(config)#router bgp 65001
R2(config-router)#neighbor 172.16.1.1 route-map ASPATH out
BB1#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 10.1.1.1/32 10.1.3.2 0 65003 65001 ?
* 172.16.1.2 11 0 65001 65001 65001 ?