Pages

Network Troubleshooting tools

With discards, the situation is almost the opposite. The packets were received with no errors but were dumped before being passed on to a higher layer protocol. A typical cause of discards is when the router needs to regain some buffer space. In the case of discards, the issue is almost always with the router that's reporting the discards (not witha a next hop device, bad cable, etc).

1) Host Configurations
ps [ ps -aux ] - lists which processes are running on the system
This combination shows all users' processes (-a), including those without controlling terminals (-x), in considerable detail (-u). The options -ax will provide fewer details but show more of the command-line arguments.
Use the -w option to extend the displayed information to 132 columns

top - gives a periodically updated listing of processes ranked in order of CPU usage

netstat [ netstat -a ] - reports the contents of kernel data structures related to networking.
netstat -rn - routeing table
netstat -i 1 - interface stats

lsof - produces a list of all open files

ifconfig [ ifconfig -a ] -  retrieve interface configurations

arp [ arp -a ] - The ARP table on a system maps network addresses into MAC addresses

portscan [ portscan 205.153.63.239 1 10000 -vv ]

2) Network tools

ping - The normal operation of ping relies on two specific ICMP messages, ECHO_REQUEST and ECHO_REPLY, but it may respond to ICMP messages other than ECHO_REPLY when appropriate

fping - extends ping to support multiple hosts in parallel

echoping - works by sending packets to one of several services that may be offered over TCP and UDP -- ECHO, DISCARD, CHARGEN, and HTTP. Particularly useful when ICMP messages are being blocked, echoping may work where ping fails

arping - uses ARP requests and replies instead of ICMP packets

3) Path tools
traceroute - a tool used to discover the links along a path.
Bandwidth measurements will give you an idea of the hardware capabilities of your network, such as the maximum capacity of your network.  (10Mbps, 100Mbps, 1Gbps)
Throughput measurements will help you discover what capacity your network provides in practice, i.e., how much of the maximum is actually available.

Throughput is generally an end-to-end measurement. When dealing with multihop paths, however, the bandwidths may vary from link to link

Bandwidth estimate:
- ping
- pathchar
- bing

Throughput estimate tools:
ttcp - One of the oldest bulk capacity measurement tools.
netperf
iperf - unix/windows
nuttcp - unix/windows

4) Packet captures
Packet capture software works by placing the network interface in promiscuous mode. In normal operations, the network interface captures and passes on to the protocol stack only those packets with the interface's unicast address, packets sent to a multicast address that matches a configured address for the interface, or broadcast packets. In promiscuous mode, all packets are captured regardless of their destination address.
On a few systems you may need to manually place the interface in promiscuous mode with the ifconfig command before running the packet capture software.

tcpdump [ tcpdump -i xl1 ]
related tools:
sanitize - a collection of five Bourne shell scripts that reduce or condense tcpdump trace files and eliminate confidential information,

tcpdpriv - is another program for removing sensitive information from tcpdump files,

tcpflow - This program allows you to capture individual TCP flows or sessions,

tcpshow - decodes a tcpdump trace file

tcptrace -  is an extremely powerful tcpdump file analysis tool, is capable of producing several types of output files for plotting with the X Window program xplot.

trafshow -  provides a continuous display of traffic over the network, giving repeated snapshots of traffic. It displays the source address, destination address, protocol, and number of bytes. This program would be most useful in looking for suspicious traffic or just getting a general idea of network traffic.

5)  Discovery tools:
 - IP Address Management
 - nmap - supports IP scanning. It also provides port scanning and stack fingerprinting
 - arpwatch - recording IP addresses and their corresponding MAC addresses

Device Identification"
- telnet
Stack Fingerprinting
- queso
- nmap Revisited [ nmap -O 172.16.2.230 ]

6) Device Monitoring
- snmp
NET SNMP (UCD SNMP - University of California at Davis)
- snmpget
- snmpgetnext, snmpwalk, and snmptable
- snmpset - is used to change the value of objects by sending SET_REQUEST messages
- snmptranslate -  Available OIDs are determined by the design of the agent and are described by its MIB. There are several different approaches you can take to discover the contents of a MIB.
snmptranslate system.sysContact.0
snmptranslate -On .1.3.6.1.2.1.1.4.0
snmptranslate -Td system.sysContact - get extended information

 Non-snmp
 With Unix, it is possible to remotely log on to a system using telnet or ssh over a network connection and reconfigure the host

7) Network monitoring
 - mrtg
 - rrd
 - zabbix

8) Custom Packets Generators
 - hping
 - nemesis
 - netcat

9) Application level tools:
 - email
 - http
 - ftp
 - dns ( nslookup and dig )

10) Log Files and Auditing

 - syslog - can log events from your Cisco router to your Unix server. There are even a number of Windows versions available

Unfortunately, many services traditionally don't do logging, either through the syslog facility or otherwise.  If these services are started by inetd, you have a couple of alternatives. Some implementations of inetd have options that will allow connection logging.
- ntp -  is a protocol you can use to synchronize the clocks on your system
 ntpdate 205.153.60.20
 ntpq -p 172.16.2.1


11) Troubleshooting Strategies
1) Document - Before you do anything else, start documenting what you are doing. This is a real test of willpower and self-discipline
2) Collect information and identify symptoms.
3) Define the problem. Once you have a clear idea, you can begin coming to terms with the problem.
[bsd1 can't telnet to lnx1. ]
4) Identify systems or subsystems involved. As you collect information, as seen in the previous example, you will define and refine not only the nature of the problem, but also the scope of the problem. This is the step in which we divide and hopefully conquer our problem.
5) Develop a testable hypothesis
6) Develop and assess solutions.
7) Implement and evaluate your solution.





PING & TRACEROUTE detailed info.

Basic Windows/*nix Network Tools:
Tool        Description
___________________________________________
Arp         Allows viewing and editing of the Address Resolution Protocol (ARP) cache.
Hostname    Displays the host name of the computer.
Ipconfig    Displays the current IPv4/IPv6 config, manage DHCP, display/flush the DNS client resolver cache.
Netsh       Configuration tool for many network services.
Netstat     Displays protocol statistics and information on current TCP connections.
Nslookup    Performs DNS queries and displays the results.
Ping        Sends ICMP Echo or ICMPv6 Echo Request messages to test reachability.
Route       Print IPv4/IPv6 routing tables and editing of the IPv4 routing table. (or netstat -r)
Tracert     Sends ICMP v4/v6 messages to trace the network route taken by packets to a specific destination.
Pathping    Sends ICMP Echo or ICMPv6 Echo Request messages to trace the route an IPv4 or IPv6 and displays packet losses for each hop (MTR).
Telnet      Tests TCP connection establishment between two nodes.
For OS Windows:
Windows Sysinternals Tools : TCPView, Autoruns, Process Explore


Additional Network Tools:

arping - sends arp and/or ip pings to a given host (for Win hardping.exe)
dig - DNS lookup utility

Route Servers - view BGP routes on the internet, they are also useful to do traceroutes and troubleshoot networks
12.0.1.28      route-server.ip.att.net
12.123.133.124  Austin, TX
12.123.5.240    Chicago,IL
12.123.1.236    New York, NY
12.123.9.241    Washington, DC

Layer 4

hp:~# cd /usr/ports/net/ripe-whois && make install clean
hp:~# whois3 -T route -i origin AS8708 | grep 'route' |awk -F " " '{print $2}' |grep /   
route:        193.230.128.0/24
route:        193.226.94.0/24
route:        217.156.62.0/24

Layer 7
HTTP PING
[root@speedtest /usr/ports/net/http_ping]# http_ping http://www.google.com
218 bytes from http://www.google.com: 115.71 ms (50.897c/64.795r/0.018d)
218 bytes from http://www.google.com: 115.793 ms (51.263c/64.515r/0.015d)
218 bytes from http://www.google.com: 115.164 ms (50.96c/64.185r/0.019d)
218 bytes from http://www.google.com: 120.705 ms (51.07c/69.614r/0.021d)
^C
--- http://www.google.com http_ping statistics ---
4 fetches started, 4 completed (100%), 0 failures (0%), 0 timeouts (0%)
total    min/avg/max = 115.164/116.843/120.705 ms
connect  min/avg/max = 50.897/51.0475/51.263 ms
response min/avg/max = 64.185/65.7773/69.614 ms
data     min/avg/max = 0.015/0.01825/0.021 ms

DNS
DNS MOST USED Resource Records (RR):

A  - mapping a DNS domain name to an IP address
isc.org.         6465    IN      A       149.20.64.42

AAAA - mapping a DNS domain name to an IPv6
isc.org.         7200    IN      AAAA    2001:4f8:0:2::d

MX - mapping a DNS domain name to the name of a computer that exchanges or forwards mail
isc.org.         7200    IN      MX      10 mx.ams1.isc.org.
isc.org.         7200    IN      MX      10 mx.pao1.isc.org.

NS -  (Name Server) a list of authoritative DNS servers for a domain.
isc.org.                1686    IN      NS      sfba.sns-pb.isc.org.
isc.org.                1686    IN      NS      ams.sns-pb.isc.org.
isc.org.                1686    IN      NS      ord.sns-pb.isc.org.

CNAME - alias of one name to another domain name

PTR - (Pointer or reverse DNS lookup) translate IPv4/IPv6 to a CNAME.
42.64.20.149.in-addr.arpa. 3600 IN      PTR     www.isc.org.

SOA -  (Start Of Authority) specifies the DNS server providing authoritative information about an domain.
isc.org.                7191    IN      SOA     ns-int.isc.org. hostmaster.isc.org. 2011062100 7200 3600 24796800 3600

____________________  DNS TOOLS  _________________________________________


dig (Domain Information Groper) - DNS lookup utility.
The source code for dig is part of the larger ISC BIND distribution.

$ dig  isc.org
$ dig  isc.org +short                  # short output
$ dig -x 149.20.64.42 +short     #  PTR
$ dig  isc.org +trace                  #  recursive DNS lookup
    DNS server itself to make queries to other DNS servers on behalf of the client who made the original request
$ dig  isc.org MX
$ dig  isc.org NS

Default ouput (quering with OS DNS setting)
$ dig isc.org
; <<>> DiG 9.6.-ESV-R3 <<>> isc.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44945
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 2

;; QUESTION SECTION:
;isc.org.                       IN      A

;; ANSWER SECTION:
isc.org.                7200    IN      A       149.20.64.42

;; AUTHORITY SECTION:
isc.org.                6839    IN      NS      ams.sns-pb.isc.org.
isc.org.                6839    IN      NS      ord.sns-pb.isc.org.
isc.org.                6839    IN      NS      ns.isc.afilias-nst.info.
isc.org.                6839    IN      NS      sfba.sns-pb.isc.org.

;; ADDITIONAL SECTION:
ns.isc.afilias-nst.info. 86039  IN      A       199.254.63.254
ns.isc.afilias-nst.info. 86039  IN      AAAA    2001:500:2c::254

;; Query time: 205 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Jun 21 13:40:05 2011
;; MSG SIZE  rcvd: 184


nslookup - query Internet name servers interactively
   nslookup [-opt ...]             # interactive mode using default server
   nslookup [-opt ...] - server    # interactive mode using 'server'
   nslookup [-opt ...] host        # just look up 'host' using default server
   nslookup [-opt ...] host server # just look up 'host' using 'server'
C:\>nslookup
Default Server:  10.137.27.172.in-addr.arpa
Address:  172.27.137.10

TYPE A / AAAA   RR (which is default)
> isc.org
Server:  10.137.27.172.in-addr.arpa
Address:  172.27.137.10
Non-authoritative answer:
Name:    isc.org
Addresses:  2001:4f8:0:2::d
            149.20.64.42

TYPE PTR  RR
> 149.20.64.42
Server:  10.137.27.172.in-addr.arpa
Address:  172.27.137.10

Name:    www.isc.org
Address:  149.20.64.42

SELECT ANOTHER DNS THROUGH WHICH WE WILL MAKE ANSWERS
> server 8.8.8.8
Default Server:  google-public-dns-a.google.com
Address:  8.8.8.8
server  - The name or IP (8.8.8.8) is looked up using the current default dns (8.8.8.8)

> lserver 8.8.8.8
Default Server:  google-public-dns-a.google.com
Address:  8.8.8.8
lserver - The name or IP (8.8.8.8) is looked up using the original default dns (172.27.137.10)

SELECT TYPE MX FOR DOMAIN  isc.org
> set type=mx
> isc.org
Server:  10.137.27.172.in-addr.arpa
Address:  172.27.137.10

Non-authoritative answer:
isc.org MX preference = 10, mail exchanger = mx.pao1.isc.org
isc.org MX preference = 10, mail exchanger = mx.ams1.isc.org

host
- DNS lookup utility
$ host isc.org
isc.org has address 149.20.64.42
isc.org has IPv6 address 2001:4f8:0:2::d
isc.org mail is handled by 10 mx.pao1.isc.org.
isc.org mail is handled by 10 mx.ams1.isc.org.

Links:
http://technet.microsoft.com/en-us/library/bb727023.aspx