Sunday, 11 March 2012

Server Monitoring using vmstat

iostat / vmstat / mpstat are some tools to monitor server performance. 

 * iostat reports CPU, disk I/O, and NFS statistics.

 * vmstat reports virtual memory statistics.

 *  mpstat reports processors statictics.

 # vmstat

 procs -----------memory-----------------          ---swap-- ------io---- ---system-- -----cpu--
 r  b    swpd     free     buff        cache   si   so    bi    bo   in   cs us sy id wa st
 1  0  196112  75232 151884 611352   0    0     8     91   45   40 20  9 71  0  0

 vmstat output contains the following fields:

    Procs – r: Total number of processes waiting to run
    Procs – b: Total number of busy processes
    Memory – swpd: Used virtual memory
    Memory – free: Free virtual memory
    Memory – buff: Memory used as buffers
    Memory – cache: Memory used as cache.
    Swap – si: Memory swapped from disk (for every second)
    Swap – so: Memory swapped to disk (for every second)
    IO – bi:  Blocks in. i.e blocks received from device (for every second)
    IO – bo:  Blocks out. i.e blocks sent to the device (for every second)
    System –  in: Interrupts per second
    System –  cs: Context switches
    CPU – us, sy, id, wa, st: CPU user time, system time, idle time, wait time

Display active and inactive memory
# vmstat -a

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free    inact    active    si   so    bi    bo   in  cs us  sy id wa st
 1  0  196112  75236 224864  787620    0    0     8    91   45   41 20  9 71  0  0

Display number of forks since last boot

This displays all the fork system calls made by the system since the last boot.

  # vmstat -f
       500235 forks  

Execute Every x seconds
# vmstat 2
# vmstat 2 5

Display slab info
# vmstat -m

Display statistics in a table format
# vmstat -s

Display disk statistics
# vmstat -d

Display statistics for a partition
 # vmstat -p /dev/sda1

Display in MB
# vmstat -S m


server monitoring useing iostat

iostat / vmstat / mpstat are some tools to monitor server performance. 

 * iostat reports CPU, disk I/O, and NFS statistics.

 * vmstat reports virtual memory statistics.

 *  mpstat reports processors statictics.

iostat

Iostat without any argument displays information about the CPU usage, and I/O statistics about all the partitions on the system are showing:

 # iostat

Linux 2.6.18-164.el5 (cloudcc.ctechz.blogspot.com)     03/11/2012

avg-cpu:  %user   %nice  %system %iowait  %steal   %idle
                  1.98     17.92    8.59       0.33       0.00       71.17

Device:            tps      Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda                13.89        33.54         365.15       14323873   155932064
sda1              13.84        32.34         363.73       13808779   155327336
sda2               0.05         1.20           1.42           514566       604728



Display only cpu statistics

# iostat -c

Linux 2.6.18-164.el5 (cloudcc.ctechz.blogspot.com)     03/11/2012

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                 1.98     17.92    8.59       0.33         0.00     71.17

Display only disk I/O statistics

# iostat -d

Linux 2.6.18-164.el5 (cloudcc.ctechz.blogspot.com)     03/11/2012

Device:         tps      Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              13.89        33.53         365.04     14323921  155949184
sda1             13.84        32.32        363.62     13808827  155344456
sda2              0.05         1.20          1.42          514566      604728

Display only network statistics

-n, displays only the device and NFS statistics.

# iostat -n

Linux 2.6.18-164.el5 (cloudcc.ctechz.blogspot.com)     03/11/2012

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                  1.98    17.93    8.59        0.33        0.00      71.17

Device:            tps     Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda                 13.89       33.52        365.00     14323937    155957552
sda1               13.84       32.32        363.58     13808843    155352824
sda2                0.05        1.20          1.42         514566        604728

Device:    rBlk_nor/s   wBlk_nor/s   rBlk_dir/s   wBlk_dir/s   rBlk_svr/s 
                                                                               wBlk_svr/s    rops/s    wops/s

192.168.1.4:/home/pac  90.67        0.00         0.00         0.00         5.33         0.00
192.168.1.4:/bkp            8.74         0.00         0.00         0.00         8.74         0.00
192.168.1.8:/media          0.02         0.00         0.00         0.00         0.01         0.00

Display I/O data in MB/second

By default iostat, displays the device I/O statistics in Blocks. To change it to MB, use -m as shown  

# iostat -m

Linux 2.6.18-164.el5 (cloudcc.ctechz.blogspot.com)     03/11/2012

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                  1.98     17.93    8.59     0.33          0.00     71.16

Device:           tps    MB_read/s   MB_wrtn/s    MB_read    MB_wrtn
sda               13.88     0.02            0.18              6994           76165
sda1             13.83     0.02            0.18              6742           75869
sda2              0.05      0.00            0.00               251             295

Display I/O statistics only for a device

# iostat -p sda

Display timestamp information

# iostat -t

Linux 2.6.18-164.el5 (cloudcc.ctechz.blogspot.com)     03/11/2012

Time: 05:51:24 PM
avg-cpu:  %user   %nice  %system %iowait  %steal   %idle
               1.98     17.93    8.59      0.33        0.00     71.16

Device:         tps      Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda              13.88        33.49        364.77        14324049   156006600
sda1             13.83        32.29        363.35        13808955   155401872
sda2              0.05         1.20          1.41           514566      604728

Execute Every x seconds - y number of times

# iostat 2
# iostat 2 3 --- execute every 2 seconds for 3 times

Display LVM statistic

# iostat -N

Mysqlslap for mysql load testing

Mysqlslap is a Load Emulation Client. mysqlslap is a diagnostic program designed to emulate client load for a MySQL server and to report the timing of each stage. It works as if multiple clients are accessing the server. mysqlslap is available as of MySQL 5.1.4.

mysqlslap runs in three stages:

1. Create schema, table, and optionally any stored programs or data you want to using for the test. This stage uses a single client connection.

2. Run the load test. This stage can use many client connections.

3. Clean up (disconnect, drop table if specified). This stage uses a single client connection.

Please find expalnations for mysqlslap options below.

# /usr/local/mysql/bin/mysqlslap -uroot -p --create-schema=DBName --query=file.txt --concurrency=100 --iterations=1 --delimiter=";"

mysqlslap usage

To apply auto generated queries,
# mysqlslap -uroot -p --auto-generate-sql


To use multiple simultaneous client connections,
# mysqlslap -uroot -p --auto-generate-sql --concurrency=100

To run the above command multiple times, iteration option is used,
# mysqlslap -uroot -p --auto-generate-sql --concurrency=100 --iterations=5

To specify the number of queries each client executes,
# mysqlslap -uroot -p --auto-generate-sql --concurrency=100 --number-of-queries=1000


The default behavior of mysqlslap when using the –auto-generate-sql switch is to create a two-column table with one integer column and one character column. To adjust these settings to include more integer and/or character columns, use the --number-char-cols and --number-int-cols switches.
# mysqlslap -uroot -p --auto-generate-sql --concurrency=100 --number-of-queries=1000 --number-char-cols=4 --number-int-cols=7


To specify the storage engine --engine option is used,
# mysqlslap -uroot -p --auto-generate-sql --concurrency=100 --number-of-queries=1000 --number-char-cols=4 --number-int-cols=7 --engine=innodb


We can also apply custom queries to mysqlslap either from a text file or directly from command line using --query option. In order to use an existing database use the --create-schema option. This doesnt create a schema but uses an existing schema.
# mysqlslap -uroot -p --create-schema=Cherry_1H_B-01 --query=cherry_1h.txt --concurrency=100 --iterations=5 --delimiter=";"



Saturday, 10 March 2012

Mysql Master Slave Replication

Setting the Replication Master Configuration

 On a replication master, you must enable binary logging and establish a unique server ID.

Binary logging must be enabled on the master because the binary log is the basis for sending data changes from the master to its slaves.

If binary logging is not enabled, replication will not be possible.

Each server within a replication group must be configured with a unique server ID.This ID is used to identify individual servers within the group.

To configure the binary log and server ID options, you will need to shut down your MySQL server and edit the my.cnf or my.ini file.  Add the following options to the configuration file within the [mysqld] section.

to enable binary logging using a log file name prefix of mysql-bin, and configure a server ID of 1, use these lines:

[mysqld]
log-bin=mysql-bin
server-id=1

After making the changes, restart the server.

If you omit server-id (or set it explicitly to its default value of 0), a master refuses connections from all slaves.

For the greatest possible durability and consistency in a replication setup using InnoDB with transactions, you should use innodb_flush_log_at_trx_commit=1 and sync_binlog=1 in the master my.cnf file.

Ensure that the skip-networking  option is not enabled on your replication master. If networking has been disabled, your slave will not able to communicate with the master and replication will fail.

if your master is also a slave (DB1 is the master of DB2, DB2 is the master of DB3) in order for DB2 to log updates from his master DB1 to the binlog (so DB3 can read them) you need to put "log-slave-updates" on my.cnf or my.ini.

Replication Implementation

 Replication is based on the master server keeping track of all changes to its databases (updates, deletes, and so on) in its binary log.

The binary log serves as a written record of all events that modify database structure or content (data) from the moment the server was started. Typically, SELECT  statements are not recorded because they modify neither database structure nor content.

Each slave that connects to the master requests a copy of the binary log. That is, it pulls the data from the master, rather than the master pushing the data to the slave. The slave also executes the events from the binary log that it receives.This has the effect of repeating the original changes just as they were made on the master. Tables are created or their structure modified, and data is inserted, deleted, and updated according to the changes that were originally made on the master.

Because each slave is independent, the replaying of the changes from the master's binary log occurs independently on each slave that is connected to the master. In addition, because each slave receives a copy of the binary log only by requesting it from the master, the slave is able to read and update the copy of the database at its own pace and can start and stop the replication process at will without affecting the ability to update to the latest database status on either the master or slave side.
  http://dev.mysql.com/doc/refman/5.0/en/replication-implementation-details.html

Masters and slaves report their status in respect of the replication process regularly so that you can monitor them.
  http://dev.mysql.com/doc/refman/5.0/en/thread-information.html

The master binary log is written to a local relay log on the slave before it is processed. The slave also records information about the current position with the master's binary log and the local relay log.
 http://dev.mysql.com/doc/refman/5.0/en/slave-logs.html

Database changes are filtered on the slave according to a set of rules that are applied according to the various configuration options and variables that control event evaluation.
   http://dev.mysql.com/doc/refman/5.0/en/replication-rules.html

Steps For Mysql Master-slave Replication

One of the biggest advantages to have master-slave set up in MySQL is to be able to use master for all of the inserts and send some, if not all, select queries to slave. This will most probably speed up your application without having to diving into optimizing all the queries or buying more hardware.

Do backups from slave: One of the advantages people overlook is that you can use MySQL slave to do backups from. That way site is not affected at all when doing backups. This becomes a big deal when your database has grown to multiple gigs and every time you do backups using mysqldump, site lags when table locks happen.You can even stop slave MySQL instance and copy the var folder instead of doing mysqldump.

let us dive into how to setup master-slave replication under MySQL. There are many configuration changes you can do to optimize your MySQL set up.

Master server ip: 192.168.1.231
Slave server ip : 192.168.1.232

Slave username : mysqlslave
Slave password : slavepwd
 
Your data directory is:  /var/lib/mysql

1. In Master

# yum install mysql mysql-server
# service mysqld start
# mysqladmin -uroot password 'master'
# service mysqld stop

Edit the my.cnf file under [mysqld] section of your mysql master

# vim /etc/my.cnf

[mysqld]
server-id = 1
relay-log = /var/lib/mysql/mysql-relay-bin
relay-log-index = /var/lib/mysql/mysql-relay-bin.index
log-error = /var/lib/mysql/mysql.err
master-info-file = /var/lib/mysql/mysql-master.info
relay-log-info-file = /var/lib/mysql/mysql-relay-log.info
datadir=/var/lib/mysql
log-bin = /var/lib/mysql/mysql-bin

# service mysqld restart

2. In Slave

# yum install mysql mysql-server
# service mysqld start
# mysqladmin -uroot password 'slave'
# service mysqld stop

Add the the following under [mysqld] to the mysql slave by edting my.cnf

# vim /etc/my.cnf

[mysqld]
server-id = 2
relay-log = /var/lib/mysql/mysql-relay-bin
relay-log-index = /var/lib/mysql/mysql-relay-bin.index
log-error = /var/lib/mysql/mysql.err
master-info-file = /var/lib/mysql/mysql-master.info
relay-log-info-file = /var/lib/mysql/mysql-relay-log.info
datadir=/var/lib/mysql

# service mysqld restart

3. Then in Mysql Master server create a user with replication privileges

# mysql -uroot -pmaster

mysql>  grant replication slave on *.* to mysqlslave@'192.168.1.232' identified by 'slavepwd';
mysql> flush privileges;

You can see the new user on the master db by

mysql> show databases;
mysql> use mysql;
mysql> show tables;
mysql> select * from user;

4. Take a dump of data from Mysql Master to move to slave

# mysqldump -uroot -p --all-databases --single-transaction --master-data=1 > masterdump.sql (master dump is put into slave inorder to make the master and slave data similar before starting the sync)

after taking the dump of master import it in slave server,

# scp masterdump.sql root@192.168.1.232:
 
# ssh root@192.168.1.232 -----> slave machine

# mysql -uroot -p < masterdump.sql

After dump is imported go in to mysql client by typing mysql. Let us tell the slave which master to connect to and what login/password to use: ( In slave machine ) 

# mysql -uroot -p
 
mysql> change master to master_host='192.168.1.231', master_user='mysqlslave', master_password='slavepwd';
 
mysql> flush privileges;

5. Let us start the slave 

mysql> start slave;

  You can check the status of the slave by typing
mysql> show slave status; 
mysql> show slave status\G
 
     ****************** 1. row *******************  
          Slave_IO_State: Waiting for master to send event
                Master_Host: 192.168.1.231
                Master_User: mysqlslave
                Master_Port: 3306
              Connect_Retry: 60
            Master_Log_File: mysql-bin.000001
        Read_Master_Log_Pos: 706
             Relay_Log_File: mysql-relay-bin.000008
              Relay_Log_Pos: 446
      Relay_Master_Log_File: mysql-bin.000001
           Slave_IO_Running: Yes
          Slave_SQL_Running: Yes
            Replicate_Do_DB:
        Replicate_Ignore_DB:
         Replicate_Do_Table:
     Replicate_Ignore_Table:
    Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
                 Last_Errno: 0
                 Last_Error:
               Skip_Counter: 0
        Exec_Master_Log_Pos: 706
            Relay_Log_Space: 446
            Until_Condition: None
             Until_Log_File:
              Until_Log_Pos: 0
         Master_SSL_Allowed: No
         Master_SSL_CA_File:
         Master_SSL_CA_Path:
            Master_SSL_Cert:
          Master_SSL_Cipher:
             Master_SSL_Key:
      Seconds_Behind_Master: 0
  1 row in set (0.01 sec)

The last row tells you how many seconds its behind the master. Don’t worry if it doesn’t say 0, the number should be going down over time until it catches up with master (at that time it will show Seconds_Behind_Master: 0) If it shows NULL, it could be that slave is not started (you can start by typing: start slave).

6. For Checking whether replication is working or not create a db in mysql master server

# create database replication;
   and check in slave whether its replicated or not
 
# show databases;
  the db replication will be there in slave as well


some NRPE Commands

You can test some of these, by running the following commands:

NRPE Commands

# /usr/local/nagios/libexec/check_nrpe -H localhost -c check_users
# /usr/local/nagios/libexec/check_nrpe -H localhost -c check_load
# /usr/local/nagios/libexec/check_nrpe -H localhost -c check_hda1
# /usr/local/nagios/libexec/check_nrpe -H localhost -c check_total_procs
# /usr/local/nagios/libexec/check_nrpe -H localhost -c check_zombie_procs

Some of the nagios commands

command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10

command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20

[check_hda1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1

[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z

command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 150 -c 200

[check_disk]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/sda

command[check_mem]=/usr/local/nagios/libexec/check_mem 85 95


Use of Nagios
 
Monitoring of network services (SMTP, POP3, HTTP, NNTP, ICMP, SNMP, FTP, SSH)

Monitoring of host resources (processor load, disk usage, system logs) on a majority of network operating systems, including Microsoft Windows with the NSClient++ plugin or Check_MK.

Monitoring of anything else like probes (temperature, alarms...) which have the ability to send collected data via a network to specifically written plugins

Monitoring via remotely-run scripts via Nagios Remote Plugin Executor.

Remote monitoring supported through SSH or SSL encrypted tunnels.

Simple plugin design that allows users to easily develop their own service checks depending on needs, by using the tools of choice (shell scripts, C++, Perl, Ruby, Python, PHP, C#, etc.)

Plugins available for graphing of data (Nagiosgraph, PNP4Nagios, Splunk for Nagios, and others available)

Parallelized service checks available

Ability to define network host hierarchy using "parent" hosts, allowing detection of and distinction between hosts that are down and those that are unreachable
 
Contact notifications when service or host problems occur and get resolved (via e-mail, pager, SMS, or any user-defined method through plugin system)

Ability to define event handlers to be run during service or host events for proactive problem resolution

Automatic log file rotation

Support for implementing redundant monitoring hosts

Optional web-interface for viewing current network status, notifications, problem history, log files, etc.

Data storage is done in text files rather than database.

 




How to add a host to Nagios server


Adding Linux host to nagios server

Following flow summarizes the above explanation:

Nagios Server (check_nrpe) —–> Remote host (NRPE deamon) —–> check_disk

Nagios Server (check_nrpe) <—– Remote host (NRPE deamon) <—– check_disk (returns disk space usage)

1. How to add Linux client to the Nagios server

To start , In the remote host add nagios user,

# /usr/sbin/useradd nagios
# passwd nagios

We need two packages. Nagios-plugin and Nrpe
---> nagios-plugins-1.4.13.tar.gz ---> nrpe-2.12.tar.gz

# cd /home/nagios/nagios-plugins-1.4.15
# ./configure
# make
# make install


The permissions on the plugin directory and the plugins will need to be fixed at this point, so run the following commands.

# chown nagios.nagios /usr/local/nagios
# chown -R nagios.nagios /usr/local/nagios/libexec

2.Install XINETD
# yum install xinetd

Install the NRPE ( Nagios Remote Plugin Executor ) daemon
Nagios Remote Plugin Executor (NRPE) is a Nagios agent that allows remote systems monitoring using scripts that are hosted on the remote systems. It allows for monitoring resources such as disk usage, system load or number of users currently logged in. Nagios periodically polls the agent on the remote system using the check_nrpe plugin.

The NRPE-plugin is running as a xinetd-service.

# wget http://osdn.dl.sourceforge.net/sourceforge/nagios/nrpe-2.12.tar.gz
# cd /home/nagios/nrpe-2.12/
# ./configure
# make all

Install the NRPE plugin (for testing), daemon, and sample daemon config file.

# make install-plugin
# make install-daemon
# make install-daemon-config


Install the NRPE daemon as a service under xinetd
# cd /home/nagios/nrpe-2.12/
# make install-xinetd

Edit the /etc/xinetd.d/nrpe file and add the IP address of the monitoring server to the only_from directive.

# vim /etc/xinetd.d/nrpe
only_from = 127.0.0.1 192.168.1.67  ------- nagios monitoring server ip

Add the following entry for the NRPE daemon to the /etc/services file.

# vim /etc/services
nrpe 5666/tcp   # NRPE

Restart the xinetd service.
# service xinetd restart


Test the NRPE daemon locally
Its time to see if things are working properly. Make sure the nrpe daemon is running under xinetd.

# netstat -at | grep nrpe

 The output out this command should show something like this:

 tcp 0 0 *:nrpe *:* LISTEN


3. Now back to your Nagios Server, and Install NRPE deamon 

# cd /home/nagios/nrpe-2.12/
# ./configure
# make all

Install the NRPE plugin

# make install-plugin

check of nrpe is working we will check the nrpe in the remote host


# /usr/local/nagios/libexec/check_nrpe -H 192.168.1.240  
 in server machine, give the ip of remote machines, to check whether they are communicating or not.

NRPE v2.12


Now in the nagios server : Go to

# cd /usr/local/nagios/etc/objects/

# cp localhost.cfg remotehost.cfg
# mv localhost.cfg Nagios-Server.cfg


Open remotehost.cfg and edit the hostname and IP adress 

# vim remoteost.cfg
define host{
  use      linux-server ; Name of host template to use
                   ; This host definition will inherit all

                   ; variables that are defined
                   ; in (or inherited by) the linux-server

                   ; host template definition.
   host_name   localhost
   alias       localhost
   address     192.168.1.240
        }

save and exit


# cd /usr/local/nagios/etc
# vim nagios.cfg

# Definitions for monitoring the local (Linux) host  --------> add and comment lines below this
#cfg_file=/usr/local/nagios/etc/objects/localhost.cfg
cfg_file=/usr/local/nagios/etc/objects/remotehost.cfg


 save and exit

edit remotehost.cfg file 

# cd /usr/local/nagios/etc/objects/
# vim /usr/local/nagios/etc/objects/remotehost.cfg


# Define a host for the local machine

define host{
         use        linux-server
         host_name   cloudcc.ctechz.blogspot.com
         alias       localhost
         address     127.0.0.1
       }


define host{
        use         linux-server           
                                                      
                                                      
        host_name    www.ctechz.com
        alias        ctechz
        address      192.168.1.240
        }


# Define an optional hostgroup for Linux machines

define hostgroup{
     hostgroup_name  linux-servers
     alias           Linux Servers
   members    cloudcc.ctechz.blogspot.com,www.ctechz.com
        } 


# SERVICE DEFINITIONS

define service{
     use        local-service       
     host_name   cloudcc.ctechz.blogspot.com,www.ctechz.com
     service_description  PING
     check_command   check_ping!100.0,20%!500.0,60%
        }


Add the Rest of the services like this 

now check if you nagios server nagios can start without errors

#  /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

If no Errors found while running above command then restart the nagios.

# service nagios restart


http://192.168.1.67/nagios/
Uname : nagiosadmin
Passwd: redhat