CCNP Switch - Multilayer Switching

To be "up/up," a router VLAN interface (SVI) must fulfill the following general conditions:
  •The VLAN exists and is "active" on the VLAN database of the switch.
  •The VLAN interface exists on the router and is not administratively down.
  •At least one Layer 2 (access port or trunk) port exists, has a link "up" on this VLAN and is in spanning-tree forwarding state on the VLAN.  

 - ip routing - enable IP routing on MLS
 - The ARP throttling adjacency drops the first packet in any communication between Hosts1 and host2 due to ARP processing when no ARP entries exist for Hosts1 and Host2.

Bridge Group Virtual Interface (BVI) -

Multilayer switching (MLS) is performed at Layers 3 and 4.
The actual MLS process can take two forms: interVLAN routing and Cisco Express Forwarding (CEF).

InterVLAN Routing
To transport packets between VLANs, you must use a Layer 3 device. Traditionally, this has been a router’s function. The router must have a physical or logical connection to each VLAN so that it can forward packets between them.

 InterVLAN Routing Connections
An interface is either in Layer 2 or Layer 3 mode, depending on the use of the command switchport
Whenever you see the word switchport, think Layer 2. So if switchport is disabled, it must be Layer 3.
Switch# show interface <type mod/num> switchport
Switch# show interface gigabitethernet 0/1 switchport
Name: Gi0/1
Switchport: Disabled
Layer 2 Port Configuration
If an interface is in Layer 3 mode and you need to reconfigure it for Layer 2 functionality instead
Switch(config)# interface type <mod/num>
Switch(config-if)# switchport          <---puts the port in Layer 2 mode
Layer 3 Port Configuration
Physical switch ports also can operate as Layer 3 interfaces, where a Layer 3 network address is assigned and routing can occur
Switch(config)# interface type <mod/num>
Switch(config-if)# no switchport
Switch(config-if)# ip address ip-address mask [secondary]
SVI Port Configuration
On a multilayer switch, you also can enable Layer 3 functionality for an entire VLAN on the switch.
SVI itself has no physical connection to the outside world. The logical Layer 3 interface is known as an SVI (vlan interface).
Switch(config)# interface vlan <vlan-id>
Switch(config-if)# ip address <ip-address mask> [secondary]
Switch(config-if)# no shutdown
The VLAN must be defined and active on the switch before the SVI can be used. 

Cisco Switching Methods
A Cisco router can use one of following three methods to forward packets:
■ Process Switching: In process switching, the router strips off the Layer 2 header for each incoming frame, looks up the Layer 3 destination network address in the routing table for each packet, and then sends the frame with rewritten Layer 2 header, including computed cyclical redundancy check (CRC), to the outgoing interface. All these operations are done by software running on the CPU for each individual frame. Process switching is the most CPU-intensive method available in Cisco routers. It can greatly degrade performance and is generally used only as a last resort or during troubleshooting.
■ Fast Switching: After the lookup of the first packet destined for a particular IP network, the router initializes the fast-switching cache used by the Fast switching mode.
When subsequent frames arrive, the destination is found in this fast-switching cache. The frame is rewritten with corresponding link addresses and is sent over the outgoing interface.
■ Cisco Express Forwarding (CEF): The default-switching mode. CEF is less CPU-intensive than fast switching or process switching. A router with CEF enabled uses in formation from tables built by the CPU, such as the routing table and ARP table, to build hardware-based tables known as the Forwarding Information Base (FIB) and adjacency tables. These tables are then used to make hardware-based forwarding decisions for all frames in a data flow, even the first. Although CEF is the fastest switching mode, there are limitations, such as other features that are not compatible with CEF or rare instances in which CEF functions can actually degrade performance, such as CEF polarization in a topology using load-balanced Layer 3 paths.

A Layer 3 switch makes forwarding decisions using one of following two methods, which are platform-dependent:
■ Route caching: Also known as flow-based or demand-based switching, a Layer 3 route cache is built within hardware functions as the switch sees traffic flow into the switch. This is functionally equivalent to Fast Switching in Router IOS.
■ Topology-based switching: Information from the routing table is used to populate the route cache, regardless of traffic flow. The populated route cache is called the FIB. CEF is the facility that builds the FIB. This is functionally equivalent to CEF in Router IOS
Because of this increased performance, topology-based switching is currently the predominate method of switching versus route caching.

CEF can occur at following two different locations on the switch:
■ Centralized switching: Carries out forwarding decisions on a specialized ASIC that is central to all interfaces of a Layer 3 switch. With centralized switching, routing, ACL,
QoS, and forwarding decisions are made on the supervisor engine in a modular chassis or by Layer 3 engines in fixed-port density Layer 3 switches. As a result, all
frames to be routed or switched must pass through the centralized engine via a fabric or bus. Furthermore, with centralized switching, the hardware-switching performance
of the Catalyst switch is based on the central forwarding engine and the fabric or bus architecture.
Examples: Catalyst 4500 / Catalyst 6500 family of switches without the use of Distributed Forwarding Cards (DFC).
■ Distributed switching: Interfaces or line modules on Layer 3 switches handle for warding decisions independently. With distributed switching, a centralized switching
engine synchronizes Layer 3 forwarding, routing, and rewrite tables to local tables on distributed switching-capable modules. As a result, individual line cards or ports
make forwarding decisions without the aid of the centralized switching engine; frames pass between ports directly across the fabric. In other words, switches using
distributed switching place additional copies of the CEF FIB and adjacency table on line modules or interfaces for routing and switching of frames.
Distributed forwarding enables Catalyst switches to achieve rates of more than 100 million pps. (Catalyst 6500 Switch Fabric module or with a Supervisor 720)

CEF is composed from:
 - FIB: Derived from the IP routing table, When the CEF TCAM table is full, a wildcard entry redirects frames to the Layer 3 engine.
 - AT: The adjacency table is derived from the ARP table and contains Layer 2 header rewrite (MAC) information for each next hop contained in the FIB.
When the adjacency table is full, a CEF TCAM table entry points to the Layer 3 engine to redirect the adjacency.

Multilayer Switching with CEF
Catalyst switches can use several methods to forward packets based on Layer 3 and Layer 4 information.

Traditionally, switches were handling Layer 2 frames at hardware (ASIC assisted) speed, whereas routers were handling  Layer 3 packets at software speed (done in CPU).
Today,  technology  enables  multilayer  switches to route packets at hardware speed.

Because a switch offloads a significant portion of the normal software-based routing process (packet rewrite) to the hardware, a multilayer switch processes more  packets, faster than a traditional router by using specialized ASIC hardware instead of  microprocessor-based engines.

Traditional MLS Overview
Multilayer switching began as a dual effort between a route processor (RP) and a switching engine (SE). The basic idea is to “route once and switch many.”
To participate in multilayer switching, the SE must know the identity of each RP. 

NetFlow switching (or route cache switching) - If the SE can switch the packet in both directions, it can learn a “shortcut path” so that subsequent packets of the same flow can be switched directly to the destination port without passing through the RP.

NetFlow switching was performed on Cisco hardware, such as:
 - Catalyst 6000 Supervisor 1/1a and Multilayer Switch Feature Card (MSFC),
 - Catalyst 5500 with a Route Switch Module (RSM), Route Switch Feature Card (RSFC), or external router.

CEF Overview
Cisco developed CEF for its line of routers, offering high performance packet forwarding through the use of dynamic lookup tables.
CEF runs by default, taking advantage of the specialized hardware

CEF also has been carried over to the Catalyst switching platforms. The following platforms all perform CEF in hardware:
■ Catalyst 6500 Supervisor 720 (with an integrated MSFC3)
■ Catalyst 6500 Supervisor 2/MSFC2 combination
■ Catalyst 4500 Supervisor III, IV, V, and 6-E
■ Fixed-configuration switches, such as the Catalyst 3750, 3560, 3550, and 2950

Hardware/Software CEF (from
Different platforms use CEF but only MLS or high end routers have dedicated hardware for performing CEF switching in hardware.
 - Hardware CEF (ASIC) on MLS like C6500/C3560, routers like ASR 1000, 12000 or CRS.
 - Software CEF (CPU) routers like C1800, C2800, C3800 up to C7200.

The process of creating the CEF table and to update it when something changes is performed in software in all cases by the routing processor.
The CEF table may be sent to modules that have a local complete copy of the CEF table this is called distributed CEF and provides the greatest performance and scalability.

The C6500 is interesting because it can mix centralized CEF modules with distributed CEF modules (those mounting a DCEF daughter card)
A GSR or CRS works only in distributed CEF mode.
C6500 architecture as an example of an high end platform

 - CEF is enabled on all CEF-capable Catalyst switches by default.
 - Catalyst 6500 (with a Supervisor 720 and its integrated MSFC3, or a Supervisor 2 and MSFC2 combination) runs CEF inherently, so CEF never can be disabled.
 - Catalyst 3750 and 4500 run CEF by default, but you can disable CEF on a per-interface basis.(no ip route-cache cef and no ip cef interface configuration commands to disable CEF on the Catalyst 3750 and 4500, respectively).
 - You should always keep CEF enabled whenever possible, except when you need to disable it for debugging purposes

A CEF-based multilayer switch consists of two basic functional blocks:
- the Layer3 engine is involved in building routing information that
- the Layer3 forwarding engine can use to switch packets in hardware.
Packet Flow Through a CEF-Based Multilayer Switch
FIB - Forwarding Information Base
The Layer 3 engine (essentially a router) maintains routing information, whether from static routes or dynamic routing protocols.
 - The new format is called a Forwarding Information Base (FIB) and contains routing or forwarding information that the network prefix can reference.
 - FIB could contain a route to along with routes to and, if those exist.
 - The FIB also contains the next-hop address for each entry.
 - FIB also contains host route (subnet mask entries (Host routes are maintained in the FIB for the most efficient routing lookup to directly connected or adjacent hosts). - When the Layer 3 engine sees a change in the routing topology, it sends an update to the FIB.

Display FIB table entries related to a specific interface or VLAN
Switch# show ip cef   [<type mod/num> | vlan <vlan-id>] [detail]
Switch# show ip cef vlan 101
Prefix           Next Hop    Interface      attached    Vlan101    Vlan101    Vlan101
FIB entries can br viewed by specifying an IP prefix address and mask
Switch# show ip cef  [<prefix-ip prefix-mask>] [longer-prefixes] [detail]
Switch# show ip cef longer-prefixes
Prefix          Next Hop      Interface     attached      Vlan101      Vlan101      Vlan101   Vlan99
         Vlan99   Vlan99
         Vlan99   Vlan99
Displaying Detailed CEF Entry Information
version number - number of times the CEF entry has been updated since the table was generated
epoch number - he number of times the CEF table has been flushed and regenerated as a whole
Switch# show ip cef detail, version 270, epoch 0, per-destination sharing
0 packets, 0 bytes
      via, Vlan99, 0 dependencies
      traffic share 1
      next hop, Vlan99
      valid adjacency
via, Vlan99, 0 dependencies
      traffic share 1
      next hop, Vlan99
      valid adjacency
0 packets, 0 bytes switched through the prefix
tmstats: external 0 packets, 0 bytes
         internal 0 packets, 0 bytes
Packets then are marked as “CEF punt” and immediately are sent to the Layer 3 engine for further processing:
 ■ An entry cannot be located in the FIB.
 ■ The FIB table is full.
 ■ The IP Time-To-Live (TTL) has expired.
 ■ The maximum transmission unit (MTU) is exceeded, and the packet must be fragmented.
 ■ An Internet Control Message Protocol (ICMP) redirect is involved.
 ■ The encapsulation type is not supported.
 ■ Packets are tunneled, requiring a compression or encryption operation.
 ■ An access list with the log option is triggered.
 ■ A Network Address Translation (NAT) operation must be performed (except on the Catalyst 6500 Supervisor 720, which can handle NAT in hardware)

CEF also can be optimized through the use of specialized forwarding hardware, using the following techniques:
■ Accelerated CEF (aCEF)—CEF is distributed across multiple Layer 3 forwarding engines (Catalyst 6500 line cards). The net result is that CEF is accelerated
on the line cards, but not necessarily at a sustained wire-speed rate.
■ Distributed CEF (dCEF)—CEF can be distributed completely among multiple Layer 3 forwarding engines for even greater performance. Because the FIB is selfcontained for complete Layer 3 forwarding, it can be replicated across any number of independent Layer 3 forwarding engines.
The Catalyst 6500 has line cards that support dCEF, each with its own FIB table and forwarding engine. A central Layer 3 engine (the MSFC3, for example) maintains the routing table and generates the FIB, which is then dynamically downloaded in full to each of the line cards.

Adjacency Table
A router normally maintains
 - a routing table containing Layer 3 network and next-hop information,
 - an ARP table containing Layer 3 to Layer 2 address mapping.
These tables are kept independently.
 - Portion of the FIB called the adjacency table, consisting of the MAC addresses of nodes that can be reached in a single Layer 2 hop.
 - The adjacency table information is built from the ARP table. 
 - As a next-hop address receives a valid ARP entry, the adjacency table is updated.
 - If an ARP entry does not exist, the FIB entry is marked as “CEF glean.” (Layer 3 forwarding engine can’t forward the packet in hardware because of the missing Layer 2 next-hop address)
If Host1 (IP= send packet to Host2 (IP= on the same network  for the first time, Host1 doesnt have MAC-IP record in ARP table. So CEF mark this adjacency as glean (wich means it needs L3 processing to find out the MAC for IP

The ARP throttling adjacency drops the first packet in any communication between Hosts1 and host2 due to ARP processing when no ARP entries exist for Hosts1 and Host2.

glean gli:n/ verb - obtain (information) from various sources, often with difficulty.

Display the adjacency table
Switch# show adjacency [<type mod/num> | vlan <vlan-id>] [summary | detail]
Switch# show adjacency summary
Adjacency Table has 24 adjacencies
Table epoch: 0 (24 entries at this epoch)
Interface Adjacency Count
Vlan99    21
Vlan101   3
SW-4500#sh adjacency  detail
Protocol Interface                 Address
IP       Vlan168         
                                   0 packets, 0 bytes
                                   epoch 0
                                   sourced in sev-epoch 57479
                                   Encap length 14
 MAC-->                    00070EC990030012DAA36A3F0800   <---IP
Displaying Adjacencies in the CEF Glean State
Switch# show ip cef adjacency glean
Prefix        Next Hop   Interface   attached   Vlan101   attached   EOBC0/0
Switch# show ip arp
Switch# show ip cef detail, version 688, epoch 0, attached, connected
0 packets, 0 bytes
    via Vlan101, 0 dependencies
        valid glean adjacency

ARP throttlingor throttling adjacency.
During the time that an FIB entry is in the CEF glean state waiting for the ARP resolution, subsequent packets to that host are immediately dropped so that the input queues do not fill and the Layer 3 engine does not become too busy worrying about the need for duplicate ARP requests
 - If an ARP reply is not received in 2 seconds, the throttling is released so that another ARP request can be triggered.
 - Otherwise, after an ARP reply is received, the throttling is released, the FIB entry can be completed, and packets can be forwarded completely in hardware.

Adjacency types
There are several types of adjacencies. Some are listed below:
■ Cache adjacency: This type of entry contains the correct outbound interface and the correct MAC address for its FIB entry. The MAC address is the IP address's MAC address if the destination's subnet is directly connected to the router, or is the MAC address of the router that the packet needs to be sent to if the destination's subnet is not directly connected to the router currently processing the packet.
■ Receive adjacency: This type of entry handles packets whose final destinations include the router itself. This includes packets whose IP addresses are assigned to the router itself, broadcast packets, and multicasts that have set up the router itself as one of the destinations.
■ Null adjacency: Handles packets destined to a NULL interface (null0 - logical interface that silently discard packets without actually forwarding them).
■ Punt adjacency: Deals with packets that require special handling or can not be switched by CEF. Such packets are forwarded to the next switching layer (generally fast switching) where they can be forwarded correctly. Are sent to the Layer 3 engine for further processing.
Switch# show cef not-cef-switched
CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp’ted Redirect    Receive Options Access Frag
RP  3579706        0          0        0   41258564       0      0    0
  - No_adj—An incomplete adjacency
  - No_encap—An incomplete ARP resolution
  - Unsupp’ted—Unsupported packet features
  - Redirect—ICMP redirect
  - Receive—Layer 3 engine interfaces; includes packets destined for IP addresses that are assigned to interfaces on the Layer 3 engine, IP network addresses, and IP broadcast addresses
  - Options—IP options present
  - Access—Access list evaluation failure
  - Frag—Fragmentation failure
■ Glean adjacency: This adjacency is created when the router knows that either the destination IP's subnet is directly connected to the router itself and it does not know that destination device's MAC address, or the router knows the IP address of the router to forward a packet to for a destination, but it does not know that router's MAC address. Packets that trigger this entry will generate an ARP request.
■ Drop adjacency: Used to switch packets that can’t be forwarded normally. In effect, these packets are dropped without being forwarded. Packets can be dropped because of an encapsulation failure, an unresolved address, an unsupported protocol, no valid route present, no valid adjacency, or a checksum error. You can gauge drop adjacency activity with the following command:
Switch# show cef drop
CEF Drop Statistics
Slot Encap_fail Unresolved Unsupported No_route No_adj ChkSum_Err
RP      8799327          1       45827  5089667    32           0
■ Discard adjacency: Used when packets must be discarded because of an ACL or other policy action.

Frame/Packet Rewrite
When a multilayer switch finds valid entries in the FIB and adjacency tables, a packet is almost ready to be forwarded.
One step remains: The packet header information must be rewritten.
 - The switch has an additional functional block that performs a packet rewrite in real time.
 - A traditional router normally would make the same changes to each packet.
 - MLS can do this very efficiently with dedicated packet-rewrite hardware and address information obtained from table lookups

When  packets  transit  through  a  router  or  multilayer  switch,  the  following  verifications must occur:
1  The incoming frame checksum is verified to ensure that no frame corruption or alteration occurs during transit.
2  The incoming IP header checksum is verified to ensure that no packet corruption or alteration occurs during transit.
Packet IN
IP unicast packets are rewritten on the output interface
1  The destination MAC address changes from the router MAC address to the next-hop MAC address.
2  The source MAC address changes from the sender MAC address to the outgoing router MAC address.
3  The destination MAC address changes from the router MAC address to the next-hop MAC address.
4  The TTL is decremented by one, and as a result, the IP header checksum is recalculated.
5  The frame checksum must be recalculated.

Packet OUT
The switch obtains the information needed for the frame rewriting process from internal tables such as CAM and TCAM tables.
Some of these tables are cached in ASICs or RAM.

CAM and TCAM Tables
Multilayer switches deploy these memory tables using specialized memory architectures, referred to as content addressable memory (CAM), and ternary content addressable memory (TCAM).
The following summarizes the tables:
■ CAM table: The primary table used to make Layer 2 forwarding decisions. The table is built by recording the source MAC address and inbound port of all incoming  frames. When a frame arrives at the switch with a destination MAC address of an en try in the CAM table, the frame is forwarded out through only the port that is associated with that specific MAC address.
■ TCAM table: Stores ACL, QoS, and other information generally associated with Layer 3 and up layer processing.
In specific high-end switch platforms, the TCAM is a portion of memory designed for rapid, hardware-based table lookups of Layer 3 and Layer 4 information. In the TCAM, a single lookup provides all Layer 2 and Layer 3 forwarding information for frames, including CAM and ACL information.

Distributed Hardware Forwarding
Each interface module includes a microcoded processor that handles all packet forwarding.
Following are the main functions of the control layer between the routing protocol and the firmware datapath microcode:
■ Managing the internal data and control circuits for the packet-forwarding and control functions
■ Extracting the other routing and packet-forwarding-related control information from the Layer 2 and Layer 3 bridging and routing protocols and the configuration  data, and then conveying the information to the interface module fo r control of the data path
■ Collecting the data path information, such as traffic statistics, from the interface module to the route processor
■ Handling certain data packets sent from the Ethernet interface modules to the route processor

CEF-Based MLS Load Sharing
CEF does support load sharing (equal-cost or nonequal-cost).
However, Hardware CEF does not support all the load-sharing features found in software-based CEF. With the current version of  software on a Catalyst 6500 switch, a single FIB entry may have up to six adja cencies for load sharing per destination.

Verifying Multilayer Switching
To verify the configuration of a Layer 2 port
Switch# show interface <type mod/num> switchport
Verify the configuration of an SVI
Switch# show interface vlan <vlan-id>
Display the IP-related information about a switch interface
Switch# show ip interface vlan 101
Vlan101 is up, line protocol is up
Internet address is
Broadcast address is
Address determined by setup command
MTU is 1500 bytes
See a summary listing of the Layer 3 interfaces involved in routing IP traffic
Switch# show ip interface brief
View the entire FIB
Switch# show ip cef
Prefix                Next Hop        Interface            receive     <---default route, no next-hop (marked “receive”)-> packets go to L3 engine for further processing      attached        Vlan1    <- “attached” because it is connected directly to an SVI, VLAN 1      receive                  <-no adjacency, and the entry is marked “receive”      receive                  <-host.Packets destined for the VLAN1 interface must be dealt with internally   Vlan1    <-entry for a neighboring multilayer switch, found on the VLAN 1 interface    receive                  <-broadcast address

Using DHCP with a Multilayer Switch
 - Dynamic Host Configuration Protocol (DHCP) is usually provide a means for dynamic address assignment to any host that can use.
 - DHCP is defined in RFC 2131 and is built around a client/server model (client request IP <-> server assign free IP to client).
 - DHCP is a dynamic mechanism, IP addresses are offered on a leased basis
 - DHCP is designed to work within a broadcast domain (DHCP server need to be located in the same broadcast domain as the client)

Configuring an IOS DHCP Server
Switch(config)# ip dhcp excluded-address <start-ip> <end-ip>
Switch(config)# ip dhcp pool <pool-name>
Switch(config-dhcp)# network <ip-address> <subnet-mask>
Switch(config-dhcp)# default-router <ip-address> [ip-address2] [ip-address3] ...
Switch(config-dhcp)# lease {infinite | {<days> [<hours> [<minutes>]]}}
Switch(config-dhcp)# exit
 - By definition, the network and broadcast addresses for the subnet won’t be offered to any client.
 - By default, leases are offered with a 1 day limit.
 - monitor the DHCP server address leases with the show ip dhcp binding

Configuring a DHCP Relay
If a DHCP server is centrally located in the network, you can configure the multilayer switch to relay DHCP messages between clients and the server, even if they are located on different VLANs or subnets.
 - As a DHCP relay, the switch will intercept the broadcast DHCP messages from the client and will forward them on to the server address as unicast messages.
 - The switch keeps track of the subnet where the client messages arrived so that it can relay the DHCP server responses back appropriately.
 - You can configure more than one helper address by repeating the ip helper-address command with different addresses (request sent simultaneously to each IP).
Switch(config)# interface vlan5
Switch(config-if)# ip address
Switch(config-if)# ip helper-address
Switch(config-if)# exit