Pages

FreeBSD step 2 - System information and monitoring

FreeBSD system information and monitoring command, tools and recommandations.

For long term monitoring = SNMP with storage MRTG/RRDTool (e.g. Cacti).
Realtime monitoring with history of any kind of information = zabbix.

A computer has four basic resources:
1) CPU
2) Memory (RAM)
3) I/O (input/output) - HDD
4) Network

If any one of them is filled to capacity, the others cannot be used to their maximum.
Performance problems are usually caused by running more tasks than the computer can handle.

Key Steps:

1) Know your hardware and resources limitation
    CPU, Memory (RAM), Disk (HDD), Network, Processes
    tools: top, vmstat, iostat, netstat, ps, ifconfig and other detailed (sar, mpstat, dstat)  
2) Know your software and configuration limitation
    OS (kernel), packages
3) know your application (MySQL)
    MySQL (error, slow query logs, mem usage)

0) Configure SNMP for FreeBSD server

1) Install bsnmp-ucd, which is a module for bsnmpd (FreeBSD snmp agent) which allows you to get memory, load average, cpu utilization and other system statistics. It implements parts of UCD-SNMP-MIB for this. 
# cd /usr/ports/net-mgmt/bsnmp-ucd && make all install clean
2) Edit /etc/snmpd.config to add the line
begemotSnmpdModulePath."ucd" = "/usr/local/lib/snmp_ucd.so"
3) Edit /etc/snmpd.config to change the read variable to set your snmp community string
Read the manual pages bsnmpd(1) and bsnmp-ucd(8) for more info.

4) Add to /etc/rc.conf
bsnmpd_enable="YES"
5) Start bsnmp by issuing
# /etc/rc.d/bsnmpd start

1) System (hardware) related information

dmesg -- display the system message buffer, startup info
$ dmesg -a | more
sysctl -- get or set system information
# sysctl -a  # all parameters
# sysctl -a | grep hw.model
# sysctl -ad  # description

sysctl -a | egrep -i 'hw.machine|hw.model|hw.ncpu'
hw.machine: amd64
hw.model: Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz
hw.ncpu: 24
hw.machine_arch: amd64

sysctl -a | egrep -i "hw.*mem:"
hw.physmem: 8569384960
hw.usermem: 7562637312
hw.realmem: 9126801408
pciconf -l -cv = diagnostic utility for the PCI bus (Show PCI devices)

sysinfo -- utility used to gather system configuration information
# cd /usr/ports/sysutils/sysinfo
# make install clean
# sysinfo -a
dmidecode -  System Information
#dmidecode -t system
 dmidecode 2.10
SMBIOS 2.5 present.

Handle 0x0100, DMI type 1, 27 bytes
System Information
Manufacturer: Dell Inc.
Product Name: PowerEdge 2950
...
Temperatures
CPU
# kldload coretemp
# echo 'coretemp_load="YES"' >> /boot/loader.conf
# sysctl -a |grep temper
HDD

a) diskinfo -ctv /dev/ad0  =  utility prints out information about a disk device, and  optionally runs a naive performance test on the device.

b) smartd is a daemon that monitors the Self-Monitoring, Analysis and Reporting Technology (SMART) system built into many ATA-3 and later ATA, IDE and SCSI-3 hard drives. The purpose of SMART is to monitor the reliability of the hard drive and predict drive failures, and to carry out different types of drive self-tests.

c) RAID Info
stats# dmesg -a | grep -i "lsi"
mpt0: <LSILogic 1030 Ultra4 Adapter> port 0x2000-0x20ff mem 0xdd210000-0xdd21ffff,0xdd200000-0xdd20ffff irq 18 at device 5.0 on pci3
da0: <LSILOGIC 1030          IM 1000> Fixed Direct Access SCSI-2 device
RAID LSI Info from /var/log/meassages:
Mar  5 17:54:33 stats kernel: mpt0:vol0(mpt0:0:0): Status ( Enabled Re-Syncing )
Mar  5 17:54:33 stats kernel: mpt0:vol0(mpt0:0:0): Low Priority Re-Sync
Mar  5 17:54:33 stats kernel: mpt0:vol0(mpt0:0:0): 1967431 of 143347995 blocks remaining
Mar  5 18:02:51 stats kernel: (mpt0:vol0:1): Physical Disk Status Changed
Mar  5 18:02:51 stats kernel: (mpt0:vol0:1): Volume Status Changed
Mar  5 18:02:51 stats kernel: mpt0:vol0(mpt0:0:0): RAID-1 - Optimal
Mar  5 18:02:51 stats kernel: mpt0:vol0(mpt0:0:0): Status ( Enabled )
Mar  5 18:02:51 stats kernel: (mpt0:vol0:1): Online
Mar  5 18:02:51 stats kernel: (mpt0:vol0:1): Status ( )
stats# mptutil show drives
mpt0 Physical Drives:
   0 (   68G) ONLINE <SEAGATE ST373207LC 7808> SAS bus 0 id 0
   1 (   68G) ONLINE <SEAGATE ST373207LC 7808> SAS bus 0 id 1
stats# mptutil show volumes
mpt0 Volumes:
  Id     Size    Level   Stripe  State  Write-Cache  Name
     0 (   68G) RAID-1          OPTIMAL   Disabled 
mptutil   show volumes 

cd /usr/ports/sysutils/smartmontools
smartctl -i /dev/ad0
smartctl -d ata -A /dev/ad0 | grep -i temperature

/usr/ports/sysutils/healthd
mbmon


2) OS info

uname = get the kernel version (and BSD version)
$ uname -a
FreeBSD myhost.servers.eu 7.1-RELEASE FreeBSD 7.1-RELEASE #0: Thu Jan  1 14:37:25 UTC 2009     root@logan.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  i386
rcorder -- print a dependency ordering of interdependent files
rcorder /etc/rc.d/* /usr/local/etc/rc.d/*


uptime = Uptime + load average
$ uptime
 1:06PM  up 42 days,  2:39, 1 user, load averages: 0.05, 0.06, 0.02
date = display or set date and time

last =  indicate last logins of users and ttys


shutdown -r now = restart system (#shutdown -r 09:30 "Reboot for maintenance")

vmstat -z   = system limit, what is necessary to increase

Check if IPFW is enabled
# sysctl -a | grep ip.fw.enable
net.inet.ip.fw.enable: 0
SSH Sessions
w = display who is logged in and what they are doing

[root@zabbix /usr/]# w
10:19AM  up 153 days, 19:12, 3 users, load averages: 1.42, 1.04, 0.82
USER       TTY      FROM                      LOGIN@  IDLE WHAT
user1   pts/0    10.11.12.51            Tue04PM 5days su (bash)
user2   pts/3    10.10.23.43            10:05AM     - mc
user3   pts/2    10.11.2.6             28Nov13 11days -bash (bash)

Drop SSH session
pkill -9 -t pts/0

Parameters:
netstat -m
= mbuf clusters in use (need to be min 10% free)
sysctl kern.maxfiles kern.openfiles   = Open files
sysctl kern.ipc.maxsockets kern.ipc.numopensockets   = Open sockets
sysctl vm.kvm_size vm.kvm_free    = KVM Size
sysctl kern.ipc.maxpipekva kern.ipc.pipekva    = Pipe KVA usage
sysctl kern.ipc.nsfbufs kern.ipc.nsfbufsused     =  Number of sendfile(2) sf_bufs in use

Configure NTP
Network Time Protocol (NTP) is a networking protocol for clock synchronization between computer systems over packet-switched, variable-latency data networks.
NTP uses a hierarchical, semi-layered system of levels of clock sources. Each level of this hierarchy is termed a stratum and is assigned a layer number starting with 0 (zero) at the top.
Stratum 0 - as atomic (caesium, rubidium) clocks, GPS clocks or other radio clocks.
Stratum 1 - serveres attached to Stratum 0 devices. Normally they act as servers for timing requests from Stratum 2 servers via NTP.
Stratum 2 - These are computers that send NTP requests to Stratum 1 servers.
Stratum 3 - These computers employ exactly the same NTP functions of peering and data sampling as Stratum 2, and can themselves act as servers for lower strata.

1)    /etc/ntp.conf
server 0.europe.pool.ntp.org
server 1.europe.pool.ntp.org
server 2.europe.pool.ntp.org
server 3.europe.pool.ntp.org
server 0.ro.pool.ntp.org
server 1.ro.pool.ntp.org
server 2.ro.pool.ntp.org
server 3.ro.pool.ntp.org
server 0.freebsd.pool.ntp.org iburst maxpoll 9
server 1.freebsd.pool.ntp.org iburst maxpoll 9
server 2.freebsd.pool.ntp.org iburst maxpoll 9
logfile /var/log/ntp.log
2)    /etc/rc.conf
ntpd_enable="YES"
ntpd_sync_on_start="YES"
3)  /etc/newsyslog.conf
/var/log/ntp.log 644 3 100 * J
4) /etc/rc.firewall
# allow ntp updates
$cmd 00307 allow udp from any 123 to external_ip in via external_interface
$cmd 00308 allow udp from external_ip to any 123 out via external_interface
5)  Restart syslog, ntp, ipfw demons
/etc/rc.d/ipfw  restart  
/etc/rc.d/syslogd restart &&  /etc/rc.d/ntpd restart
6) Verify
# ntpq -p     
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+svp.provu.co.uk 213.2.4.80       4 u   14  128  377   85.478    9.648  18.087
-procyon.promode 147.231.100.5    3 u    9  128  377   41.975  -12.456  26.108
-sysprod.libfy.c 192.93.2.20      2 u   90  128  377   63.729   44.592  31.613
+ns1.luns.net.uk 157.44.176.4     2 u   16  128  377   68.927   -9.027  30.874
+89.149.54.30    131.107.13.100   2 u   13  128  377   12.901   -9.513  31.644
+birlic.alsys.ro 80.96.120.252    2 u   17  128  377   47.640   10.853  26.184
+ups.alsys.ro    192.53.103.108   2 u    7  128  377   47.632   19.362  22.653
+cache.alsys.ro  192.53.103.108   2 u   15  128  377   12.864   36.949  32.341
+chomper.fedora. 193.226.65.36    2 u   66  128  377    0.811   28.456  28.943
+jekader.fedora. 129.69.1.153     2 u   12  128  377    0.404   -0.122  22.713
*95-65-29-212.st 80.96.120.253    2 u  125  128  377    0.989   -1.793  20.938
 * - current NTP server selected
# cat /var/log/ntp.log
25 Jul 08:41:29 ntpd[12004]: synchronized to 212.121.88.250, stratum 2
25 Jul 11:47:34 ntpd[12004]: time reset +11165.244720 s
25 Jul 11:50:53 ntpd[12004]: synchronized to 80.68.242.188, stratum 3

7) Manual NTP update
 ntpdate <ntp server ip> = update system time
# ntpdate 0.europe.pool.ntp.org
11 May 11:52:59 ntpdate[29677]: adjust time server 213.198.55.2 offset 0.000356 sec

3) Basic OS health overview

http://freebsdguide.ru/_19/_3/

Terms used by unix systems for Memory management:
Virtual Memory - The entire memory space available to the Unix Kernel. This consists of both the physical memory and all swap space.
Physical memory - is the RAM attached to the motherboard. This is the stuff you go out and buy and fit to your computer.  It does NOT include CPU cache, video memory or other memory attached to controller cards etc.
Swap space (a.k.a Paging space) - The space on disk (HDD) used to extend the physical memory of a system. Swap space is a how a system with 64MB Ram can run a 100MB program.With rare exceptions all Unix systems should have at least one Swap space
Paging - The action of transferring one Page of physical memory from, or to, the swap space. A page is typically in the order of 4KB.
Swapping - The action of transferring an entire program (code & data) between Physical memory and the Swap space.Note that very few Unix systems ever swap in practice.

vmstat - report virtual memory statistics
vmstat 3    = Refresh stats every "3" seconds
vmstat -z    = Report on memory used by the kernel zone allocator
[sc@core ~]$ vmstat 1
 procs      memory      page                      disks     faults           cpu
 r b w     avm    fre   flt  re  pi  po  fr  sr   mf0 mf1   in   sy    cs    us sy id
 1 40 0   2696M   285M  2498  31  74  6  299 116   0   0    1038  130  194   12 5  84
16 47 0   2824M   403M  2527  0   0   0  603 556  10  55    1090 9445  18865 29 10 61
procs
   r - The number of processes that are blocked waiting for CPU time. If this number is high, your CPU is the bottleneck.
   b - The number of processes that are blocked waiting for system input/output (generally, waiting for disk access). If this number is high, your disk is the bottleneck.
   w - The number of processes that are runnable but are entirely swapped out.If you regularly have processes swapped out, your memory is inadequate for the work you are doing on the system.
memory
  avm - The average number of pages of virtual memory that are in use. If this value is abnormally high or increasing, your system is actively consuming swap space.
  fre - The number of memory pages available for use. If this value is abnormally low, you have a memory shortage.
page
   flt - The number of page faults, where information needed was not in real memory and had to be fetched from swap space or disk.
   re - The number of pages that have been reclaimed or reused from cache.
   pi - Short for pages in; this is the number of pages being moved from real memory to swap.
   po - Short for pages out; this is the number of pages being moved from swap to real memory.
   fr - How many pages are freed per second.
   sr - How many pages are scanned per second.
disks
 The number shown is the number of disk operations per second, a valuable clue to determining how well your disks are handling their load.
faults
Faults are not bad; they’re just received system traps and interrupts.
   in - The number of system interrupts (IRQ requests) received in the last five seconds.
   sy - The number of system calls in the last five seconds.
   cs - The number of context switches in the last second, or a per-second average since the last update.
cpu
Breakdown of percentage usage of CPU time.
us - user time for normal and low priority processes
sy - system tasks
id - cpu idle

systat = display system statistics
systat [-display] [refresh-interval]
systat -vmstat 1     = Display general system load
systat -netstat 1     = Display network connections.
systat -ifstat 1     = Display the network traffic going through active interfaces
systat -tcp 1         = TCP statistics (also: icmp, ip, icmp6, ip6 )
systat -iostat 1     = Display statistics about processor use and disk throughput
systat -swap 1         = Show information about swap space usage on all the swap areas compiled into the kernel
RAM memory
fetch http://pastebin.com/download.php?i=0mCKp5gn;   tr -d '\015' < download.php\?i=0mCKp5gn  > free; mv free /usr/local/bin/free; chmod +x /usr/local/bin/free
$  free
SYSTEM MEMORY INFORMATION:
mem_wire:         484429824 (    461MB) [ 15%] Wired: disabled for paging out
mem_active:  +    217874432 (    207MB) [  7%] Active: recently referenced
mem_inactive:+   1957052416 (   1866MB) [ 62%] Inactive: recently not referenced
mem_cache:   +     34160640 (     32MB) [  1%] Cached: almost avail. for allocation
mem_free:    +    412622848 (    393MB) [ 13%] Free: fully available for allocation
mem_gap_vm:  +       835584 (      0MB) [  0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all:     =   3106975744 (   2963MB) [100%] Total real memory managed
mem_gap_sys: +     94896128 (     90MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    =   3201871872 (   3053MB)        Total real memory available
mem_gap_hw:  +     19353600 (     18MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      =   3221225472 (   3072MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:         817389568 (    779MB) [ 25%] Logically used memory
mem_avail:   +   2403835904 (   2292MB) [ 74%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   =   3221225472 (   3072MB) [100%] Logically total memory

Swap Information
pstat, swapinfo -- display system data structures
See how large system tables have become if the system is under heavy load
swapinfo = swap information
# swapinfo -h
Device          1K-blocks     Used    Avail Capacity
/dev/ad0s1b       1013632     116K     990M     0%
#pstat -T
332/7880 files
0M/1485M swap space

# sysctl vm.stats.vm.v_swappgsout
vm.stats.vm.v_swappgsout: 12

# sysctl vm.stats.vm.v_swappgsin
vm.stats.vm.v_swappgsin: 11

4) CPU and Input/Output monitoring

Performance metrics for CPU:
1) Cpu Utilization
  - This indicates how much of the CPU is currently getting used.
  - This is fairly straight forward, and you can view the CPU utilization from the top command.
  - 100% CPU utilization means the system is fully loaded.
  - So, a higher %age of CPU utilization will cause performance issues.

2) Context Switch
  - When CPU switches from one process (or thread) to another, it is called as context switch.
  - Context switching is very essential for multitasking of the CPU.
  - However, a higher level of context switching can cause performance issues.

3) Run Queue
  - Run queue indicates the total number of active processes in the current queue for CPU.
  - When CPU is ready to execute a process, it picks it up from the run queue based on the priority of the process.
  - Please note that processes that are in sleep state, or i/o wait state are not in the run queue.
  - So, a higher number of processes in the run queue can cause performance issues.

4) Load Average
  - This indicates the average CPU load during the last one-, five-, and fifteen-minute periods.
  - Metric = 1, CPU's run queue is full.
  - Max metric = number of CPU/cores (4 cores = load avg 4 shows system is 100% load)
  - Calculated by combining both the total number of process in the queue, and the total number of processes in the uninterruptable task status.

top = realtime display information about the top cpu processes and of what are they doing:
   - paging to/from swap,
   - spending lots of time in the kernel, or processing interrupts,
   - processes/threads are using CPU (default view)
   - processes/threads are using I/O (top -m io -o total) = Linux iotop analog for FreeBSD,
   - syscalls (STATES):
RUN, CPU0, CPU1...  - proccess is executed
ucond/umtx: waiting on an application thread lock(Это может обозначать, что поток ждёт выполнения другого занятого потока).
biord/biowr/wdrain - disk I/Oactivity

sbwait (Wait for data to arrive at/drain from a socket buffer) - perfectly normal, legal, state. Daemons get into it because they're waiting for anything to happen on the socket(s) they are listening to. Process is waiting for a socket buffer to be filled or emptied.
nanslp - Process is blocked in the nanosleep(2) syscall, which is an explicit request to pause for a certain amount of time. Hard to generalize this one. It should be infrequent,
or it might be used by something implementing its own polling.
select - Process is blocked in the select(2) syscall. Обычно это значит, что процесс ждёт внешнего ввода (сеть, терминал, изменение файла, и т.п.). select поддерживает только ожидание чего-то, что связано с файловым дескриптором (например socket, tty, vnode).
lockf - Process is blocked waiting for a file lock to be released. Could be an flock(2) lock or an fcntl(2)/F_SETLK record lock.
piperd - Process is waiting for data to arrive on a pipe ("rd" is short for read).

I/O View:
voluntary context switch - process blocks waiting for a resource
involuntary context switch - kernel decides that the process should stop running for now

Processes:
ps
= process status
-a      Display information about other users' processes as well as your own.
-x      When displaying processes matched by other options, include processes which do not have a controlling terminal.
-w      Use 132 columns to display information, instead of the default  which is your window size. 
          If the -w option is specified more than once, ps will use as many columns as necessary without regard for your window size.
-u      Display information associated with the following keywords: user, pid, %cpu, %mem, vsz, rss, tt, state, start, time, and command.
         The -u option implies the -r option.         

-m      Sort by memory usage, instead of the combination of controlling terminal and process ID.
-r      Sort by current CPU usage, instead of the combination of controlling terminal and process ID.

-t      Display information about processes attached to the specified terminal devices. ("w" command will help)

sample:
# ps -auxwww | grep mysql
mysqld1  86615  0.0 10.3 119828 52748  p8  I    11:00AM   1:20.02 [mysqld]
another method:
Top CPU processes: ps auxwww|sort -nr +2| grep -v idle| head -15
Top Memory processes: ps auxwww|sort -nr +3| head -15


pstree = processes tree
dtpstree = display a tree of processes

# dtpstree -t
/sbin/init-+-/usr/sbin/cron
           +-/sbin/devd
           +-/usr/libexec/ftpd
           +-8*[/usr/libexec/getty]
           +-healthd
           +-/usr/local/sbin/httpd---8*[/usr/local/sbin/httpd]
           +-/usr/sbin/inetd
           +-mysqld
           +-ntpd: ntp engine---ntpd: dns engine
           +-ntpd: [priv]
           +-/usr/local/sbin/munin-node
           +-sendmail: accepting connections
           +-sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue
           +-/usr/sbin/sshd-+-sshd: sc [priv]---sshd: sc@ttyp7----bash---su----su---mc---bash---dtpstree
           |                +-sshd: sc [priv]---sshd: sc@ttyp0----bash---su----su---mc---bash
           +-/home/radiusd1/sbin/startd---/home/radiusd1/sbin/radiusd
           +-/usr/sbin/syslogd
           +-zabbix_agentd: main process-+-zabbix_agentd: main process
                                         +-3*[zabbix_agentd: waiting for connection]
                                         +-zabbix_agentd: poller [sleeping for 1 seconds]


# start ATOP fixed with 1 sec interval, per-second specific
atop -f1 1

(interactive 'l' key , CPU->0)

Disk info

gstat -- print statistics about GEOM disks
# gstat
dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0     14     12    173    5.0      2     40    0.1    6.0| mfid0
    0     14     12    173    5.0      2     40    0.1    6.0| mfid0s1
    0     69      0      0    0.0     69   8711    0.5    1.7| mfid1
    0      0      0      0    0.0      0      0    0.0    0.0| cd0
    0      0      0      0    0.0      0      0    0.0    0.0| mfid0s1a
    0      0      0      0    0.0      0      0    0.0    0.0| mfid0s1b
    0      0      0      0    0.0      0      0    0.0    0.0| mfid0s1d
    0     12     12    173    5.0      0      0    0.0    6.0| mfid0s1e
    0      2      0      0    0.0      2     40    0.1    0.0| mfid0s1f
    0     69      0      0    0.0     69   8711    0.5    1.7| mfid1s1
    0      0      0      0    0.0      0      0    0.0    0.0| md0
    0     69      0      0    0.0     69   8711    0.6    1.7| mfid1s1d

top -m io -o vcsw 

mount = shows mounted file systems with flags
df -h = display free disk space in `Human-readable' output
du -h -d 1 /usr = disk usage (folder) statistics, in `Human-readable' output, 1 depth /usr4) ncdu = NCurses Disk Usage

fdisk /dev/ad0 = PC slice table maintenance utility
disklabel /dev/ad0s1 = read and write BSD label
/stand/sysinstall = can ciew and modify disk mounting points
cat /etc/fstab = static information about the file systems

fstat = identify active files
pstat -f = display system data structures

iostat = report I/O statistics
iostat  -d  -w 5 mfid0

# iostat -dxz -w 10 -c 2        (count=2)
                        extended device statistics 
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b 
mfid0     23.0  24.7   823.1   811.9    0  18.9   8
mfid1      2.4  32.9    32.0  3864.6    0   9.8   2
                        extended device statistics 
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b 
mfid0      0.3   3.9     3.2   108.2    0   0.7   0
mfid1      0.3  22.6     4.8  2725.2    0   1.0   1 
# iostat -d 5      (update every 5 seconds)
           mfid0            mfid1              md0              cd0
  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s
 34.26  48  1.60  110.41  35  3.81   0.00   0  0.00   0.00   0  0.00
 13.96   3  0.04  115.00  13  1.44   0.00   0  0.00   0.00   0  0.00
 20.07   6  0.11  124.34  33  4.03   0.00   0  0.00   0.00   0  0.00
# iostat -x 5
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
mfid0     23.0  24.7   823.1   811.9    0  18.9   8
mfid1      2.4  32.9    31.9  3864.6    0   9.8   2
md0        0.0   0.0     0.0     0.0    0   0.4   0
cd0        0.0   0.0     0.0     0.0    0   0.0   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
mfid0      1.0   3.0    36.8    44.5    0   1.7   1
mfid1      0.0   8.0     0.0   849.9    0   0.5   0
md0        0.0   0.0     0.0     0.0    0   0.0   0
cd0        0.0   0.0     0.0     0.0    0   0.0   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
^C

systat -vmstat

Process debbuging : ktrace, truss
ktrace -i -p 5349
 ktrace -C
 kdump -Hs

5) Network Information


For Network Troubleshooting read.

ifconfig
= view and configure network interface parameters (IP, mask, MAC, status)

systat -ifstat 1
= trafic statistic for "1" seconds (OR use ifstat)

netstat
- вывод активных сетевых соединений (сокетов)
systat -netstat n   = realtime netstat

sockstat
= list open sockets
sockstat -4

trafshow -i rl0   = traffic related to interface "rl0"
iftop   = display bandwidth usage on an interface by host
nettop   = показывает сетевую активность по категориям
nstrems   =  показывает текущие потоки на интерфейсе
tcptrack    =  показывает текущие соединения

/usr/ports/net/bmon

netstat -r   = routing table
tcpflow
arp -a  = ARP table

Packet sniffers like tcpdump, netflow:
tcpdump -i rl0 host 192.168.61.20 and port 80

Tcpdump packet content
tcpdump -i em1 -nnXSs 0 'port 80'
netstat -s = errors or places where you’re out of memory or buffers.
# ifconfig network-interface down
# ifconfig network-interface up

# /etc/rc.d/netif start
# /etc/rc.d/netif stop
# /etc/rc.d/netif restart

# /etc/rc.d/routing restart
# /etc/rc.d/networking restart && /etc/rc.d/routing restart




    6) Other information
    To see the last 10 lines of a long file, use “tail filename”. To see the first 10 lines, use “head filename”.
    /var/mail/root

    /var/spool/mail/root сыплются сообщения системы об ошибках и других событиях

    tmux vs screen - terminal
    http://www.dayid.org/os/notes/tm.html


    time = The time utility executes and times the specified command.
    # time ntpdate 0.europe.pool.ntp.org
    11 May 11:54:21 ntpdate[30324]: adjust time server 213.198.55.2 offset 0.002878 sec

    real    0m0.823s
    user    0m0.000s
    sys     0m0.002s

    6) MySQL top
    cd /usr/ports/databases/mtop
    make install clean
    setup:
    For mysql 4.0.2 and greater:
        mysql> grant super, reload, process on *.* to mysqltop;
        mysql> grant super, reload, process on *.* to mysqltop@localhost;
        mysql> flush privileges;

    7)
    Compare
    diff - outputs the differences between two files

    8) pdmenu - /usr/local/etc/pdmenurc
    #!/usr/local/bin/pdmenu
    # Sample menus for Pdmenu.

    # Define the main menu.
    menu:main:Main Menu
    show:_Updates::updates
    show:_Restart Services ::restart
    show:_Monitoring ::monitoring
    nop
    show:_System Info (CPU, Mem) ::sysinfo
    nop
    exec:_Who's online?:truncate:echo "These users are online:";w -n
    exec:_Change current user's password:pause:passwd
    nop
    exec:_Pdmenu edit menu::mcedit /usr/local/etc/pdmenurc
    nop
    exit:_Exit

    menu:updates:Updates:Some regular updates
    exec:_Ports Update::portsnap fetch update
    exec:Ports First Time (_extract)::portsnap fetch extract
    nop
    exec:_FreeBSD fetch updates ::freebsd-update fetch
    exec:_FreeBSD fetch install ::freebsd-update install
    exec:_FreeBSD fetch rollback ::freebsd-update rollback
    nop
    exec:_NTPdate:pause:ntpdate 2.md.pool.ntp.org
    nop
    exit:_Main menu..

    menu:restart:Restart Services:Restart some services
    exec:_IPFW::/etc/rc.d/ipfw restart
    exec:_Netif and routing::/etc/rc.d/netif restart && /etc/rc.d/routing restart
    nop
    exec:_FreeBSD fetch updates ::freebsd-update fetch
    exec:_FreeBSD fetch install ::freebsd-update install
    exec:_FreeBSD fetch rollback ::freebsd-update rollback
    nop
    nop
    exit:_Main menu..

    menu:monitoring:Monitoring:Some basic monitoring
    exec:_Top::top
    exec:_vmstat 1::vmstat 1
    nop
    nop
    nop
    exit:_Main menu..

    menu:sysinfo:Sysinfo:Hardware system information
    exec:_OS version :display,pause:sysinfo -c os
    exec:_CPU :display,pause:sysinfo -c cpu
    exec:_RAM :display,pause:sysinfo -c mem
    exec:_Network :display,pause:sysinfo -c  network
    nop
    exec:_All :display,pause:sysinfo -c -a
    nop
    nop
    exit:_Main menu..



    Info from:
    http://216.147.18.102/unixfaq/explain_vm.shtml
    http://www.thegeekstuff.com/2011/03/linux-performance-monitoring-intro/

      1 comment: