Pages

CCNP Switch bonus - Troubleshooting Ethernet

Troubleshooting Ethernet
http://www.cisco.com/en/US/docs/internetworking/troubleshooting/guide/tr1904.html

Interface Commands (show interfaces - show interfaces vg-anylan)
http://www.cisco.com/c/en/us/td/docs/ios/12_2/interface/command/reference/finter_r/irfshoin.html

Cisco NX-OS/IOS Interface Comparison
http://docwiki.cisco.com/wiki/Cisco_NX-OS/IOS_Interface_Comparison


Options to interface commands
Switch#show interfaces fastEthernet 0/1 accounting
FastEthernet0/1
                Protocol    Pkts In   Chars In   Pkts Out  Chars Out
                   Other          0          0         40       2580
           Spanning Tree        303      18180         72       4320
                     CDP          6       1864         20       8640
                     VTP          0          0          1         99
                     DTP          6        360          0          0
Switch#show interfaces fastEthernet 0/1 capabilities
    FastEthernet0/1
      Model:                 WS-C3560-48TS
      Type:                  10/100BaseTX
      Speed:                 10,100,auto
      Duplex:                half,full,auto
      Trunk encap. type:     802.1Q,ISL
      Trunk mode:            on,off,desirable,nonegotiate
      Channel:               yes
      Broadcast suppression: percentage(0-100)
      Flowcontrol:           rx-(off,on,desired),tx-(none)
      Fast Start:            yes
      QoS scheduling:        rx-(not configurable on per port basis),
                             tx-(4q3t) (3t: Two configurable values and one fixed.)
      CoS rewrite:           yes
      ToS rewrite:           yes
      UDLD:                  yes
      Inline power:          no
      SPAN:                  source/destination
      PortSecure:            yes
      Dot1x:                 yes
SW1# show interfaces gig 0/9 counters errors
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize
Gi0/9  0  0  0  0  0
Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants
Gi0/9  5603  0  5373  0  0  0  0
Switch#show interfaces fastEthernet 0/1 transceiver properties
    Diagnostic Monitoring is not implemented.
    Name : Fa0/1
    Administrative Speed: auto
    Administrative Duplex: auto
    Administrative Auto-MDIX: on
    Administrative Power Inline: N/A
    Operational Speed: auto
    Operational Duplex: auto
    Operational Auto-MDIX: on
    Media Type: 10/100BaseTX
sw1#sh interfaces fastEthernet 1/0/1 switchport
    Name: Fa1/0/1
    Switchport: Enabled
    Administrative Mode: trunk
    Operational Mode: trunk (member of bundle Po1)
    Administrative Trunking Encapsulation: dot1q
    Operational Trunking Encapsulation: dot1q
    Negotiation of Trunking: On
    Access Mode VLAN: 1 (default)
    Trunking Native Mode VLAN: 1 (default)
    Administrative Native VLAN tagging: enabled
    Voice VLAN: none
    Administrative private-vlan host-association: none
    Administrative private-vlan mapping: none
    Administrative private-vlan trunk native VLAN: none
    Administrative private-vlan trunk Native VLAN tagging: enabled
    Administrative private-vlan trunk encapsulation: dot1q
    Administrative private-vlan trunk normal VLANs: none
    Administrative private-vlan trunk associations: none
    Administrative private-vlan trunk mappings: none
    Operational private-vlan: none
    Trunking VLANs Enabled: ALL
    Pruning VLANs Enabled: 2-1001
    Capture Mode Disabled
    Capture VLANs Allowed: ALL
    Protected: false
    Unknown unicast blocked: disabled
    Unknown multicast blocked: disabled
    Appliance trust: none
sw1#

show interface command in depth
Original source:  http://www.ciscozine.com/show-interface-in-depth/

In my opinion, a good network engineer must know the “show interface” in depth; indeed, this command is useful to obtain various interface information like drop, duplex mismatch, error, tx/rx load, …
Usually, the IOS switch/router have similar “show interface” output; the differences are dictated by devices, interface and IOS.
Below a show interface of a TenGigabitEthernet interface. The show is issued on a Cisco WS-C6509-E in VSS Mode with IOS version 15.
Ciscozine-IOS#sh int te1/5/4
TenGigabitEthernet1/5/4 is up, line protocol is up (connected)
  Hardware is C6k 10000Mb 802.3, address is 0000.0000.fd90 (bia 0008.ef4a.fd90)
  MTU 1500 bytes, BW 10000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s, media type is 10Gbase-SR
  input flow-control is on, output flow-control is off
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 7000 bits/sec, 8 packets/sec
  5 minute output rate 10000 bits/sec, 11 packets/sec
  L2 Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes
  L3 in Switched: ucast: 0 pkt, 0 bytes - mcast: 0 pkt, 0 bytes mcast
  L3 out Switched: ucast: 0 pkt, 0 bytes mcast: 0 pkt, 0 bytes
     4495527 packets input, 488522378 bytes, 0 no buffer
     Received 4460539 broadcasts (1153347 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     6925984 packets output, 825456963 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
Ciscozine-IOS#

TenGigabitEthernet1/5/4 is up, line protocol is up (connected)
Identify if the interface is phisically up and if the protocol is up.
Hardware is C6k 10000Mb 802.3, address is 0000.0000.fd90 (bia 0008.ef4a.fd90)
Identify the hardware interface and the interface mac-address; 
 - the BIA aka Burned-In (MAC) Address cannot be changed, 
 - while the “address can be changed with the command “mac-address 0000.0000.fd90″ under the interface configuration mode.
 - Remember: When the interface mac address is changed, the arp or mac address table associated to the interface will be “linked” with the custom mac address!

MTU 1500 bytes, BW 10000 Kbit/sec, DLY 100 usec

 - MTU: define the Maximum Transmission Unit. More info http://en.wikipedia.org/wiki/Maximum_Transmission_Unit
 - BW: The bandwidth command is only there to communicate the speed of the interface to higher level protocols. Most of the time, a routing protocol needs to know the speed of the interface so it can choose the best route. In the case of routing protocols, IGRP, EIGRP, and OSPF all use the bandwidth statement.
 - DLY: Propagation delay is the delay it takes for information to transmit from one point and be received by another down a line or through the air. Delay is another number used by a routing protocol to decide on the “best” route for traffic. It was intended to be the “delay” for packets over that path, so a routing protocol could choose the lowest delay path to send packets.

reliability 255/255, txload 1/255, rxload 1/255
 - reliability of the interface as a fraction of 255 (255/255 is 100 percent reliability), calculated as an exponential average over 5 minutes.
 - txload/rxload=Load on the interface as a fraction of 255 (255/255 is completely saturated), calculated as an exponential average over 5 minutes.

Encapsulation ARPA, loopback not set
Define the interface encapsulation; nowadays, you will see only the ARPA. In the past, there were also SAP and SNAP encapsulation.
 - Loopbacks are an important part of troubleshooting; they are used to isolate the fault on and end-to-end circuit (especially when the circuit is down). More info http://www.cisco.com/…/tech_note09186a00800c93c4.shtml

Keepalive set (10 sec)
Keepalives are used on the routers interfaces as hello mechanism to check the end to end connectivity to the other end.Routers interface used this mechanism to check the interface status.If you have no keepalive command its means that interface status check mechanism in disabled and router will not transmit any keepalive packet on the link.

Full-duplex, 10Gb/s, media type is 10Gbase-SR
Define the physical speed of the interface and if it works in half or duplex mode. The last part of the line defines the type of the media.

input flow-control is on, output flow-control is off
Flow-control is a mechanics allowing the receiving party of a connection to control the rate of the sending party. You may see many different implementations of flow-control technologies at different levels of OSI model (e.g. XON/XOFF for RS232, TCP sliding window, B2B credits for Fibre Channel, FECN/BECN for Frame-Relay, ICMP source-quench message, etc). More info http://blog.ine.com/2008/07/08/802-3x-flow-control

Clock mode is auto
This command is supported on the 1Gb/10Gb transceivers only.
If the clock mode of the near end of a link does not match the clock mode of the far end, the line protocol does not come up. The active and passive clock status is determined during the auto negotiation process before the transmission link is established.

ARP type: ARPA, ARP Timeout 04:00:00
ARP type defines the encapsulation type of the interface; tipically, for ethernet interface is ARPA. The default ARP timeout is 4hours but can be customized using the command “arp timeout [timeout]“

Last input never, output never, output hang never
Last input, output are the number of hours, minutes, and seconds since the last packet was successfully received or transmitted by the interface.
Note: This counter is updated only when packets are process switched, not when packets are fast switched.

Last clearing of “show interface” counters never
It  is the last time the clear counters command was issued since the last time the switch was rebooted. The clear counters command is used to reset interface statistics.

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
 - Input queue is the number of packets in the input queue.
 - Size/max/drops = the current number of frames in the queue / the max number of frames the queue can hold before it must start dropping frames / the actual number of frames dropped because the max queue size was exceeded.
 - Flushes is used to count Selective Packet Discard. SPD is a mechanism that quickly drops low priority packets when the CPU is overloaded in order to save some processing capacity for high priority packets. The flushes counter in the show interface command output increments as part of selective packet discard (SPD), which implements a selective packet drop policy on the IP process queue of the router. Therefore, it applies to only process switched traffic.
The purpose of SPD is to ensure that important control packets, such as routing updates and keepalives, are not dropped when the IP input queue is full. When the size of the IP input queue is between the minimum and maximum thresholds, normal IP packets are dropped based on a certain drop probability. These random drops are called SPD flushes.
 - Total output drops is the number of packets dropped because the output queue is full. A common cause of this might be traffic from a high bandwidth link being switched to a lower bandwidth link or traffic from multiple inbound links being switched to a single outbound link. For example, if a large amount of bursty traffic comes in on a gigabit interface and is switched out to a 100Mbps interface, this might cause output drops to increment on the 100Mbps interface. This is because the output queue on that interface is overwhelmed by the excess traffic due to the speed mismatch between the inbound and outbound bandwidths.

Queueing strategy: fifo
First-in, first-out (FIFO) queuing is the default queuing strategy that applies to all interfaces with more than 2 Mbps, or, in other words, E1 size or greater interfaces. With the FIFO Queuing strategy, packets are forwarded through the interface in the order that they are received. Other methods can be: WFQ, CBWFQ, …

Output queue: 0/40 (size/max)
The number of packets in the output queue. Size/max means the current number of frames in the queue/the max number of frames the queue can hold before it is full and must start dropping frames.

5 minute input rate 7000 bits/sec, 8 packets/sec
5 minute output rate 10000 bits/sec, 11 packets/sec
The average input and output rate seen by the interface in the last five minutes. In order to get a more accurate reading by specifying a shorter period of time (to better detect traffic bursts for example), issue the “load-interval <seconds>” interface command.

4495527 packets input, 488522378 bytes, 0 no buffer
Packets input: Total number of error-free packets received by the system.
Bytes: Total number of bytes, including data and MAC encapsulation, in the error-free packets received by the system.
No buffers: Number of received packets discarded because there was no buffer space in the main system. Compare with ignored count. Broadcast storms on Ethernet networks and bursts of noise on serial lines are often responsible for no input buffer events.

Received 4460539 broadcasts (1153347 multicasts)
Total number of broadcast or multicast packets received by the interface.

0 runts, 0 giants, 0 throttles
 
- Runts: Number of packets that are discarded because they are smaller than the minimum packet size of the medium. For instance, any Ethernet packet that is less than 64 bytes is considered a runt.
 - Giants: Number of packets that are discarded because they exceed the maximum packet size of the medium. For example, any Ethernet packet that is greater than 1518 bytes is considered a giant.
 - Throttles: the number of times the receiver on the port is disabled, possibly because of buffer or processor overload. If an asterisk (*) appears after the throttles counter value, it means that the interface is throttled at the time the command is run.
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
 
- Input error: Includes runts, giants, no buffer, CRC, frame, overrun, and ignored counts. Other input-related errors can also cause the input errors count to be increased, and some datagrams may have more than one error; therefore, this sum may not balance with the sum of enumerated input error counts.
 - CRC: Cyclic redundancy checksum generated by the originating LAN station or far-end device does not match the checksum calculated from the data received. On a LAN, this usually indicates noise or transmission problems on the LAN interface or the LAN bus itself. A high number of CRCs is usually the result of collisions or a station transmitting bad data.
 
- Frame: Number of packets received incorrectly having a CRC error and a noninteger number of octets. On a LAN, this is usually the result of collisions or a malfunctioning Ethernet device.
 - Overrun: Number of times the receiver hardware was unable to hand received data to a hardware buffer because the input rate exceeded the receiver’s ability to handle the data.
 - Ignored: Number of received packets ignored by the interface because the interface hardware ran low on internal buffers. These buffers are different than the system buffers mentioned previously in the buffer description. Broadcast storms and bursts of noise can cause the ignored count to be increased.

0 watchdog, 0 multicast, 0 pause input
 - Watchdog: Number of times watchdog receive timer expired. It happens when receiving a packet with length greater than 2048.
 - Pause input: Counter incrementing means that the port is receiving pause frame. Pause frame is a packet that tells the far-end device to stop transmitting packets until the sender is able to handle all the traffic and clear it’s buffers. It could be caused by a oversubscription of bandwidth, or a burst traffic pattern.

0 input packets with dribble condition detected
 - Dribble bit error indicates that a frame is slightly too long. This frame error counter is incremented just for informational purposes; the router accepts the frame.

6925984 packets output, 825456963 bytes, 0 underruns
 - Packets output: Total number of messages transmitted by the system.
 - Bytes: Total number of bytes, including data and MAC encapsulation, transmitted by the system.
 
- Underruns: Number of times that the transmitter has been running faster than the router can handle. This may never be reported on some interfaces.

0 output errors, 0 collisions, 1 interface resets
 - Output errors: Sum of all errors that prevented the final transmission of datagrams out of the interface being examined. Note that this may not balance with the sum of the enumerated output errors, as some datagrams may have more than one error, and others may have errors that do not fall into any of the specifically tabulated categories.
 - Collisions: Number of messages transmitted because of an Ethernet collision. A packet that collides is counted only once in output packets.
 
- Interface resets: Number of times an interface has been completely reset. This can happen if packets queued for transmission were not sent within several seconds. On a serial line, this can be caused by a malfunctioning modem that is not supplying the transmit clock signal, or by a cable problem. If the system notices that the carrier detect line of a serial interface is up, but the line protocol is down, it periodically resets the interface in an effort to restart it. Interface resets can also occur when an interface is looped back or shut down.

0 babbles, 0 late collision, 0 deferred
 
- Babbles: Babble errors occur due to the transmission of frames in excess of 1518 bytes in size.
 
- Late collision: Number of late collisions. Late collision happens when a collision occurs after transmitting the preamble. The most common cause of late collisions is that your Ethernet cable segments are too long for the speed at which you are transmitting.
 
- Deferred: Deferred indicates that the chip had to defer (postpone) while ready to transmit a frame because the carrier was asserted.

0 lost carrier, 0 no carrier, 0 PAUSE output
 - Lost carrier: Number of times the carrier was lost during transmission.
 - No carrier: Number of times the carrier was not present during the transmission.
 - PAUSE output: Pause outputs occur when the receiving port is getting overloaded and the so the device sends a pause request to the device connected to the port.

0 output buffer failures, 0 output buffers swapped out
 
- Output buffer failures: Number of failed buffers and number of buffers swapped out.
 
- Output buffers swapped outIf the outbound interface transmit queue is full, then the packet is copied from a hardware buffer to DRAM, then copied back to the transmit queue when there is room.

L2 Switched: ucast: 0 pkt, 0 bytes – mcast: 0 pkt, 0 bytes
L3 in Switched: ucast: 0 pkt, 0 bytes – mcast: 0 pkt, 0 bytes mcast
L3 out Switched: ucast: 0 pkt, 0 bytes mcast: 0 pkt, 0 bytes
The output indicates how many packets have been L2 switched on the interface as well as how many packets have been L3 switched in and out of the interface.

Remember: There is a difference between the counter of show interface command output for a physical interface and a VLAN interface.
 - The input packet counters increment in the output of show interface for a VLAN interface when that packet is Layer 3 (L3) processed by the CPU.
 - Traffic that is Layer 2 (L2) switched never makes it to the CPU and is not counted in the show interface counters for the VLAN interface. It would be counted on the show interface output for the appropriate physical interface.

In NX-OS (Nexus device) the “show interface” output is slightly different than the IOS output, but it is it is easy to understand. Below an example:
Ciscozine-NX-OS# sh interface ethernet 1/1
Ethernet1/1 is up
 Dedicated Interface
  Hardware: 1000/10000 Ethernet, address: 000d.ecdd.2fc8 (bia 000d.ecdd.2fc8)
  Description: TERADATA - F4238
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
  reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA
  Port mode is access
  full-duplex, 10 Gb/s, media type is 10G
  Beacon is turned off
  Input flow-control is off, output flow-control is off
  Rate mode is dedicated
  Switchport monitor is off
  EtherType is 0x8100
  Last link flapped 5week(s) 6day(s)
  Last clearing of "show interface" counters never
  30 seconds input rate 51128 bits/sec, 2 packets/sec
  30 seconds output rate 109088 bits/sec, 15 packets/sec
  Load-Interval #2: 5 minute (300 seconds)
    input rate 78.13 Kbps, 9 pps; output rate 113.67 Kbps, 11 pps
  RX
    22236230840 unicast packets  4414705 multicast packets  997021 broadcast packets
    22241642569 input packets  27905275144675 bytes
    881597017 jumbo packets  0 storm suppression packets
    0 runts  0 giants  3 CRC  0 no buffer
    3 input error  0 short frame  0 overrun   0 underrun  0 ignored
    0 watchdog  0 bad etype drop  0 bad proto drop  0 if down drop
    0 input with dribble  0 input discard
    0 Rx pause
  TX
    12574452594 unicast packets  112812737 multicast packets  66330588 broadcast packets
    12753595920 output packets  5123002661192 bytes
    1001411772 jumbo packets
    1 output errors  0 collision  0 deferred  0 late collision
    0 lost carrier  0 no carrier  0 babble 0 output discard
    0 Tx pause
  9 interface resets

Ciscozine-NX-OS#

Interfaces counters errors
Switch#sh interfaces FastEthernet0/5   counters errors

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards
Fa0/5               0           0           0           0          0            0

Port      Single-Col  Multi-Col   Late-Col  Excess-Col  Carri-Sen      Runts     Giants
Fa0/5              0          0          0           0          0          0          0
Switch#
Align-Err 
An alignment error occurs when frames do not end with an even number of octets, while simultaneously having a bad Cyclic Redundancy Check (CRC). An alignment error normally suggests a Layer 1 issue, such as cabling or port (either switch port or NIC port) issues.

FCS-Err 
A Frame Check Sequence (FCS) error occurs when a frame has an invalid checksum, although the frame has no framing errors. Like the Align-Err error, an FCS-Err often points to a Layer 1 issue.

Xmit-Err
A transmit error (that is, Xmit-Err) occurs when a port’s transmit buffer overflows. A speed mismatch between inbound and outbound links often results in a transmit error.

Rcv-Err
A receive error (that is, Rcv-Err) occurs when a port’s receive buffer over flows. Congestion on a switch’s backplane could cause the receive buffer on a port to fill to capacity, as frames await access to the switch’s backplane. However, most likely, a Rcv-Err is indicating a duplex mismatch. UnderSize An undersize frame is a frame with a valid checksum but a size less than  64 bytes. This issue suggests that a connected host is sourcing invalid frame sizes.

Single-Col
A Single-Col error occurs when a single collisions occurs before a port successfully transmits a frame. High bandwidth utilization on an attached link or a duplex mismatch are common reasons for a Single-Col error.

Multi-Col 
A Multi-Col error occurs when more than one collision occurs before a port successfully transmits a frame. Similar to the Single-Col error, high band width utilization on an attached link or a duplex mismatch are common reasons for a Multi-Col error.

Late-Col
A late collision is a collision that is not detected until well after the frame has begun to be forwarded. While a Late-Col error could indicate that the connected cable is too long, this is an extremely common error seen in mismatched duplex conditions.

Excess-Col
The Excess-Col error occurs when a frame experienced sixteen successive collisions, after which the frame was dropped. This error could result from high bandwidth utilization, a duplex mismatch, or too many devices on a segment.

Carri-Sen
The Carri-Sen counter is incremented when a port wants to send data on a half-duplex link. This is normal and expected on a half-duplex port, because the port is checking the wire, to make sure no traffic is present, prior to sending a frame. This operation is the carrier sense procedure described by the Carrier Sense Multiple Access with Collision Detect (CSMA/CD) operation used on half-duplex connections. Full-duplex connections, however, do not use CSMA/CD.

Runts
A runt is a frame that is less than 64 bytes in size and has a bad CRC. A runt could result from a duplex mismatch or a Layer 1 issue.

Giants
A giant is a frame size greater than 1518 bytes (assuming the frame is not a jumbo frame) that has a bad FCS. Typically, a giant is caused by a problem with the NIC in an attached host.