CCNP Switch - Aggregating Switch Links (L2/L3 Etherchannel,PAgP,LACP)

EtherChannel summary
 - configuration changes made on port-channel interface apply to ALL physical interfaces from port-channel,
 - before being bundled, each of the physical ports from channel should have the same SPEED/DUPLEX/NATIVE-VLAN/VLAN- range/TRUNK status/TYPE.
 - up to eight physical ports of the same Ethernet media type and speed,
 - Generally, all bundled ports first must belong to the same VLAN
 - If used as a trunk, bundled ports must be in trunking mode, have the same native VLAN, and pass the same set of VLANs.
 - must be configured with identical STP settings.
 - LACP does not support half-duplex. Half-duplex ports in an LACP EtherChannel are put in the suspended state.
 - Configure all LAN ports in an EtherChannel to use the same EtherChannel protocol; you cannot run two EtherChannel protocols in one EtherChannel.


L2L3 etherchannel 
 - Layer 3 EtherChannels, assign Layer 3 addresses to the port channel logical interface, not to the LAN ports in the channel.
 - Layer 2 EtherChannels, assign all LAN ports in the EtherChannel to the same VLAN or configure them as trunks.


Layer 3 EtherChannels, you must manually create the port-channel logical interface first and then put the Layer 3 LAN ports in to the channel group
Layer 2 EtherChannels, configure the LAN ports with the channel-group command, which automatically creates the port channel logical interface.
 - You cannot put Layer 2 LAN ports into a manually created port channel interface.
 - For Cisco IOS to create port channel interfaces for Layer 2 EtherChannels, the Layer 2 LAN ports must be connected and functioning.

Port channel Configuration
 - After you configure an EtherChannel, configuration changes applied to the port-channel interface apply to all the physical ports assigned to the port-channel interface.
 - Configuration changes applied to the physical port affect only the port where you apply the configuration.
 - To change the parameters of all ports in an EtherChannel, apply configuration commands to the port-channel interface, for example, STP commands or commands to configure a Layer 2 EtherChannel as a trunk.
info: http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/release/12-1_19_ea1/configuration/guide/3750scg/swethchl.pdf

Default EtherChannel Configuration
Feature                          Default Setting
Channel groups                   None assigned.
Port-channel logical interface   None defined.
PAgP mode                        No default.
PAgP learn method                Aggregate-port learning on all ports.
PAgP priority                    128 on all ports.
LACP mode                        No default.
LACP learn method                Aggregate-port learning on all ports.
LACP port priority               32768 on all ports.
LACP system priority             32768.
LACP system ID                   LACP system priority and the switch MAC address.
Load balancing                   Source-MAC address of the incoming packet.


L2 etherchannel config (catalyst 4500)
!
interface Port-channel45
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,4,9
 switchport mode trunk
 switchport nonegotiate
!
interface GigabitEthernet7/6
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,4,9
 switchport mode trunk
 switchport nonegotiate
 speed 1000
 duplex full
 no cdp enable
 channel-group 45 mode on
!
interface GigabitEthernet6/29
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,4,9
 switchport mode trunk
 switchport nonegotiate
 speed 1000
 duplex full
 no cdp enable
 channel-group 45 mode on
!

L3 etherchannel config (Cisco 7600)
!
interface Port-channel46
 ip address 10.1.1.1 255.255.255.252
!
interface TenGigabitEthernet7/1
 no ip address
 load-interval 30
 no cdp enable
 channel-group 12 mode on
!
interface TenGigabitEthernet7/2
 no ip address
 shutdown
 no cdp enable
 channel-group 12 mode on
!

Troubleshooting commands
show etherchannel summary                   EtherChannel status of each member port
show etherchannel port
show etherchannel port-channel              Time stamps of EtherChannel changes
show etherchannel detail                    Detailed status about each EtherChannel component
show etherchannel load-balance              Load-balancing hashing algorithm
show etherchannel port-channel              Load-balancing port index used by hashing algorithm
show {pagp | lacp} neighbor                 EtherChannel neighbors on each port
show lacp sys-id                            LACP system ID

Cisco offers a method of scaling link bandwidth by aggregating, or bundling, parallel links, termed the EtherChannel technology.

Two to eight links of either are bundled as one logical link of Fast EtherChannel (FEC), Gigabit EtherChannel (GEC), or 10-Gigabit Etherchannel (10GEC):
 - Fast Ethernet (FE)  full-duplex bandwidth of up to 1600 Mbps (8x100Mbps down + 8x100Mbps up),
 - Gigabit Ethernet (GE) max 16 Gbps,
 - 10-Gigabit Ethernet (10GE) max 160 Gbps.

Ordinarily, having multiple or parallel links between switches creates the possibility of bridging loops, an undesirable condition.
EtherChannel avoids this situation by bundling parallel links into a single, logical link, which can act as either an access or a trunk link.

Although an EtherChannel link is seen as a single logical link, the link doesn’t necessarily have an inherent total bandwidth equal to the sum of its component physical links. (FEC max is 800 Mbps) the single resulting FEC bundle does not operate at this speed.
Instead, traffic is distributed across the individual links within the EtherChannel.

If one link within the bundle is favored by the load-distribution algorithm, that link will carry a disproportionate amount of traffic. In other words, the load isn’t always distributed equally among the individual links.

EtherChannel also provides redundancy with several bundled physical links. If one of the links within the bundle fails, traffic sent through that link automatically is moved to an adjacent link. Failover occurs in less than a few milliseconds and is transparent to the end user.

Distributing Traffic in EtherChannel
Load is not necessarily balanced equally across all the links
Frames are forwarded on a specific link as a result of a hashing algorithm.
The hash algorithm computes a binary pattern that selects a link number in the bundle to carry each frame.
The algorithm can use:
 -  src/dst IP, or a combination of both
 -  src/dst MAC,
 - TCP/UDP port numbers.

If two addresses or port numbers are hashed, a switch performs an exclusive-OR (XOR) operation on one or more low-order bits of the addresses or TCP/UDP port numbers as an index into the bundled links.

EtherChannel consisting of
 - two links bundled together requires a 1-bit index (2^1=2 links). If the index is 0, link 0 is selected; if the index is 1, link 1 is used.
 - four-link bundle uses a hash of the last 2 bits (2^2=4).
 - eight-link bundle uses a hash of the last 3 bits (2^3=8).

Example
As an example, consider a packet being sent from IP address 192.168.1.1 to 172.31.67.46.
 - Because EtherChannels can be built from two to eight individual links, only the rightmost (least-significant) 3 bits are needed as a link index.
 - From the source and destination addresses, these bits are 001 (1) and 110 (6)
 - a two-link EtherChannel,  a 1-bit XOR is performed on the rightmost address bit: 1 XOR 0 = 1, causing Link 1 in the bundle to be used.
 - a four-link EtherChannel produces a 2-bit XOR: 01 XOR 10 = 11, causing Link 3 in the bundle to be used.
 - an eight-link EtherChannel requires a 3-bit XOR: 001 XOR 110 = 111, where Link 7 in the bundle is selected.

A conversation between two devices always is sent through the same EtherChannel link because the two endpoint addresses stay the same.
However, when a device talks to several other devices, chances are that the destination addresses are distributed equally with 0s and 1s in the last bit (even and odd address values).

Load imbalance - if one pair of hosts has a much greater volume of traffic than the other pair, one link in the channel will be used much more than the other.
To remedy this condition, you should consider other methods of hashing algorithms for the channel. For example, a method that combines the source and destination addresses along with UDP or TCP port numbers in a single XOR operation can distribute traffic much differently.
Switch(config)# port-channel load-balance <method>
method       Hash    Switch Model
src-ip       bits   All models
dst-ip       bits   All models
src-dst-ip   XOR    All models
src-mac      bits   All models
dst-mac      bits   All models
src-dst-mac  XOR    All models
src-port     bits   6500, 4500
dst-port     bits   6500, 4500
src-dst-port XOR    6500, 4500
The default configuration is to use source XOR destination IP addresses, or the src-dst-ip method.
The default for the Catalyst 2970 and 3560 is src-mac for Layer 2 switching.
If Layer 3 switching is used on the EtherChannel, the src-dst-ip method will always be used, even though it is not configurable.
SW-4500#show etherchannel port-channel
                Channel-group listing:
                ----------------------
Group: 45
----------
                Port-channels in the group:
                ---------------------------
Port-channel: Po45
------------
Age of the Port-channel   = 49d:17h:02m:47s
Logical slot/port   = 11/45          Number of ports = 2
GC                  = 0x00000000
Port state          = Port-channel Ag-Inuse
Protocol            =    -

Ports in the Port-channel:

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
  0     00     Gi6/29   On/FEC             0
  1     00     Gi7/6    On/FEC             0  <---- Although this information is not intuitive, you can use the hex values to get an idea of each link’s traffic loads relative to the others.

Time since last port bundled:    49d:17h:02m:47s    Gi7/6
Time since last port Un-bundled: 49d:17h:02m:47s    Gi7/6
* If most of the traffic is IP, it might make sense to load balance according to IP addresses or TCP/UDP port numbers.

For non-IP frames: If a frame can’t meet the load-balancing criteria, the switch automatically falls back to the “next lowest” method. With Ethernet, MAC  addresses must always be present, so the switch distributes those frames according to their MAC addresses.

A switch also provides some inherent protection against bridging loops with EtherChannels.
When ports are bundled into an EtherChannel, no inbound (received) broadcasts and multicasts are sent back out over any of the remaining ports in the channel.
Outbound broadcast and multicast frames are load-balanced like any other.

EtherChannel Negotiation Protocols
EtherChannels can be negotiated between two switches to provide some dynamic link configuration. (PAgP and LACP)
Mode                 Negotiation Packets Sent?   Characteristics
PAgP       LACP
On         On        No                          All ports channeling
Auto       Passive   Yes                         Waits to channel until asked
Desirable  Active    Yes                         Actively asks to form a channel
Both the active and passive LACP modes enable ports to negotiate with partner ports to an EtherChannel based on criteria such as port speed and, for Layer 2 EtherChannels, trunking state and VLAN numbers.

Port Aggregation Protocol
The Port Aggregation Protocol (PAgP) is a Cisco-proprietary protocol that can be run only on Cisco switches and on those switches licensed by vendors to support PAgP.

PAgP can be configured in active mode (desirable), in which a switch actively asks a far end switch to negotiate an EtherChannel, or in passive mode (auto, the default), in which a switch negotiates an EtherChannel only if the far end initiates it.

PAgP (Port Aggregation Protocol) - Cisco proprietary protocol  for automatic EtherChannel configuration and negotiation between switches.
PAgP forms an EtherChannel only on ports that are configured for either identical static VLANs or trunking.
PAgP also dynamically modifies parameters of the EtherChannel if one of the bundled ports is modified.

PAgP Interaction with Other Features
DTP and CDP send and receive packets over the physical ports in the EtherChannel.
Trunk ports send and receive PAgP protocol data units (PDUs) on the lowest numbered VLAN. 
 - In Layer 2 EtherChannels, the first port in the channel that comes up provides its MAC address to the EtherChannel. If this port is removed from the bundle, one of the remaining ports in the bundle provides its MAC address to the EtherChannel.
 - For Layer 3 EtherChannels, the MAC address is allocated by the stack master as soon as the interface is created (through the interface port-channelglobal configuration command).
 - PAgP sends and receives PAgP PDUs only from ports that are up and have PAgP enabled for the auto or desirable mode.

Link Aggregation Control Protocol
LACP is a standards-based alternative to PAgP, defined in IEEE 802.3ad.
LACP packets are exchanged between switches over EtherChannel-capable ports. LACP also assigns roles to the EtherChannel’s endpoints.

The switch with the lowest system priority (a 2-byte priority value followed by a 6-byte switch MAC address) is allowed to make decisions about what ports actively are participating in the EtherChannel at a given time.
Ports are selected and become active according to their port priority value (a 2-byte priority followed by a 2-byte port number), where a low value indicates a higher priority.
A set of up to 16 potential links can be defined for each EtherChannel. Through LACP, a switch selects up to eight of these having the lowest port priorities as active EtherChannel links at any given time. The other links are placed in a standby state and will be enabled in the EtherChannel if one of the active links goes down.

Like PAgP, LACP can be configured in active mode (active), in which a switch actively asks a far-end switch to negotiate an EtherChannel, or in passive mode (passive), in which a switch negotiates an EtherChannel only if the far end initiates it.

LACP Interaction with Other Features
The DTP and the CDP send and receive packets over the physical ports in the EtherChannel.
Trunk ports send and receive LACP PDUs on the lowest numbered VLAN.
 - In Layer 2 EtherChannels, the first port in the channel that comes up provides its MAC address to the EtherChannel. If this port is removed from the bundle, one of the remaining ports in the bundle provides its MAC address to the EtherChannel.
 - For Layer 3 EtherChannels, the MAC address is allocated by the stack master as soon as the interface is created through the interface port-channelglobal configuration command.
 - LACP sends and receives LACP PDUsonly from ports that are up and have LACP enabled for the active or passive mode.

EtherChannel Configuration
PAgP
To configure switch ports for PAgP negotiation (the default)
Switch(config)# interface <type mod/num>
Switch(config-if)# channel-protocol pagp
Switch(config-if)# channel-group <number> mode {on | {{auto | desirable} [non-silent]}}
On all Cisco IOS–based Catalyst models, you can select between PAgP and LACP as a channel-negotiation protocol.
Some older models, however, offer only PAgP, so the channel-protocol command is not available.
The interfaces are not assigned to channel groups until you configure them manually.

By default, PAgP operates in silent submode with the desirable and auto modes, and allows ports to be added to an EtherChannel even if the other end of the link is silent and never transmits PAgP packets.
 - If you expect a PAgP-capable switch to be on the far end, you should add the non-silent keyword to the desirable or auto mode.
 - The non-silent keyword is always used with the auto or desirable mode.
 - This requires each port to receive PAgP packets before adding them to a channel. If PAgP isn’t heard on an active port, the port remains in
the up state, but PAgP reports to the Spanning Tree Protocol (STP) that the port is down.
 - If you do not specify non-silent with the auto or desirable mode, silent is assumed.
 - The  silent setting is for connections to file servers or packet analyzers; this setting enables PAgP to operate, to attach the interface to a channel group, and to use the interface for transmission.

Tip: In practice, you might notice a (15-second) delay from the time the links in a channel group are connected until the time the channel is formed and data can pass over it.
Even if the two interfaces are using PAgP auto mode, the link will still eventually come up, although not as a channel. You might notice that the total delay before data can pass over the link is actually approximately 45 or 50 seconds.
The first 15 seconds are the result of PAgP silent mode waiting to hear inbound PAgP messages, and the final 30 seconds are the result of the STP moving through the listening and learning stages.

example: EtherChannel load-balancing hash of both source and destination port numbers, switch actively negotiating a channel
Switch(config)# port-channel load-balance src-dst-port
Switch(config)# interface range gig 3/1 – 4
Switch(config-if)# channel-protocol pagp
Switch(config-if)# channel-group 1 mode desirable non-silent

LACP
To configure switch ports for LACP negotiation
Switch(config)# lacp system-priority <priority>
Switch(config)# interface <type mod/num>
Switch(config-if)# channel-protocol lacp
Switch(config-if)# channel-group <number> mode {on | passive | active}
Switch(config-if)# lacp port-priority <priority>  on         Enable Etherchannel only
  active     Enable LACP unconditionally     (it will start to send LACP packets)
  passive    Enable LACP only if a LACP device is detected    (
it will wait until a LACP packet will arrive)
  desirable  Enable PAgP unconditionally    (it will start to send PAgP packets)
  auto       Enable PAgP only if a PAgP device is detected     (it will wait until a PAgP packet will arrive)
 - First, the switch should have its LACP system priority defined (1 to 65,535; default 32,768).
If not changed both switches will have the same system priority (32,768), and the one with the lower MAC address will become the decision maker.
 - lacp port-priority - configure a lower port priority (1 to 65,535; default32,768) for any interfaces that must be active, and a higher priority for interfaces that
might be held in the standby state. This prepares extra standby interfaces to replace failed active ones.

Example configuration
Switch(config)# lacp system-priority 100
Switch(config)# interface range gig 2/1 – 4 , gig 3/1 – 4
Switch(config-if)# channel-protocol lacp
Switch(config-if)# channel-group 1 mode active
Switch(config-if)# lacp port-priority 100
Switch(config-if)# exit
Switch(config)# interface range gig 2/5 – 8 , gig 3/5 – 8   <--- default 32768 lacp port-priority, so this ports will be held as stanby interfaces
Switch(config-if)# channel-protocol lacp
Switch(config-if)# channel-group 1 mode active
Troubleshooting an EtherChannel
show etherchannel summary
show etherchannel port
show running-config interface <type mod/ num>
show interface <type mod/num> etherchannel
show etherchannel load-balance

The whole concept is based on consistent configurations on both ends of the channel.
 - EtherChannel on mode does not send or receive PAgP or LACP packets.
 - EtherChannel desirable (PAgP) or active (LACP) mode attempts to ask the far end to bring up a channel. Therefore, the other end must be set to either desirable or
auto mode.
 - EtherChannel auto (PAgP) or passive (LACP) mode participates in the channel protocol, but only if the far end asks for participation. Therefore, two switches in the auto or passive mode will not form an EtherChannel.
 - PAgP desirable and auto modes default to the silent submode, in which no PAgP packets are expected from the far end. If ports are set to nonsilent submode, PAgP
packets must be received before a channel will form.

Verify the EtherChannel state:
Cisco7600#show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator
        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port
Number of channel-groups in use: 14
Number of aggregators:           14

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
11     Po11(RU)         -        Te4/2(P)    Te7/4(P)   
12     Po12(RU)         -        Te4/1(P)    Te7/1(P)  
Verify the channel negotiation mode:
Cisco7600# show etherchannel port   
                Channel-group listing:
                -----------------------
Group: 11
----------
                Ports in the group:
                -------------------
Port: Te4/2
------------
Port state    = Up Mstr In-Bndl
Channel group = 11          Mode = On               Gcchange = -
Port-channel  = Po11        GC   =   -              Pseudo port-channel = Po11
Port index    = 0           Load = 0x55             Protocol =    -
Mode = LACP
Age of the port in the current state: 170d:20h:08m:43s

Port: Te7/4
------------
Port state    = Up Mstr In-Bndl
Channel group = 11          Mode = On               Gcchange = -
Port-channel  = Po11        GC   =   -              Pseudo port-channel = Po11
Port index    = 1           Load = 0xAA             Protocol =    -
Mode = LACP
Age of the port in the current state: 157d:20h:46m:30s
Cisco7600#sh run int Te4/2
interface TenGigabitEthernet4/2
 no ip address
 load-interval 30
 channel-group 11 mode on
Cisco7600#sh int Te4/2 etherchannel
Port state    = Up Mstr In-Bndl
Channel group = 11          Mode = On               Gcchange = -
Port-channel  = Po11        GC   =   -              Pseudo port-channel = Po11
Port index    = 0           Load = 0x55             Protocol =    -
Mode = LACP
Age of the port in the current state: 170d:20h:12m:18s
Cisco7600# show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
        dst-ip
        mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Destination MAC address
  IPv4: Destination IP address
  IPv6: Destination IP address
  MPLS: Label or IP

Some messages from the switch might look like errors but are part of the normal EtherChannel process. For example, as a new port is configured as a member of an existing EtherChannel, you might see this message:
4d00h: %EC-5-L3DONTBNDL2: FastEthernet0/2 suspended: incompatible partner port  with FastEthernet0/1
When the port first is added to the EtherChannel, it is incompatible because the STP runs on the channel and the new port. After STP takes the new port through its progression of states, the port is automatically added into the EtherChannel.

Other messages do indicate a port-compatibility error (different duplex mode than the other ports in the EtherChannel)
4d00h: %EC-5-CANNOT_BUNDLE2: FastEthernet0/3 is not compatible with FastEthernet0/1 and will be suspended (duplex of Fa0/3 is full, Fa0/1 is half)