DRBD is a block device which is designed to build high availability clusters.
This is done by mirroring a whole block device via (a dedicated)
network.
DRBD takes over the data, writes it to the local disk and sends it to the other host.
On the other host, it takes it to the disk there. The other components needed are a cluster membership service, which is supposed to be heartbeat, and some kind of application that works on top of a block device. Each device (DRBD provides more than one of these devices) has a state, which can be 'primary' or 'secondary'. If the primary node fails, heartbeat is switching the secondary device into primary state and starts the application there.If the failed node comes up again, it is a new secondary node and has to synchronise its content to the primary. This, of course, will happen whithout interruption of service in the background.
The
Distributed Replicated Block Device (DRBD) is a
Linux Kernel module that constitutes a distributed storage system. You can use DRBD to share block devices between Linux servers and, in turn, share file systems and data.
DRBD implements a block device which can be used for storage and which is replicated from a primary server to one or more secondary servers. The distributed block device is handled by the DRBD service. Each DRBD service writes the information from the DRBD block device to a local physical block device (hard disk).
On the primary data writes are written both to the underlying physical block device and distributed to the secondary DRBD services. On the secondary, the writes received through DRBD and written to the local physical block device.The information is shared between the primary DRBD server and the secondary DRBD server synchronously and at a block level, and this means that DRBD can be used in high-availability solutions where you need failover support.
When used with MySQL, DRBD can be used to ensure availability in the event of a failure. MySQL is configured to store information on the DRBD block device, with one server acting as the primary and a second machine available to operate as an immediate replacement in the event of a failure.
For automatic failover support, you can combine DRBD with the Linux Heartbeat project, which manages the interfaces on the two servers and automatically configures the secondary (passive) server to replace the primary (active) server in the event of a failure.
You can also combine DRBD with MySQL Replication to provide both failover and scalability within your MySQL environment.
NOTE:- Because DRBD is a Linux Kernel module, it is currently not supported on platforms other than Linux.
Configuring the DRBD Environment
To set up DRBD, MySQL and Heartbeat, you follow a number of steps that affect the operating system, DRBD and your MySQL installation.
@ DRBD works through two (or more) servers, each called a
node.
@ Ensure that your
DRBD nodes are as identically configured as possible, so that the secondary machine can act as a direct replacement for the primary machine in the event of system failure.
@ The
node that contains the primary data, has read/write access to the data, and in an
HA environment is the currently active node is called the primary.
@ The server to which the data is replicated is called the
secondary.
@ A
collection of nodes that are sharing information is referred to as a
DRBD cluster.
@
For DRBD to operate, you must have a block device on which the information can be stored on each DRBD node. The lower level block device can be a physical disk partition, a partition from a volume group or RAID device or any other block device.
@ For the distribution of data to work, DRBD is used to create a logical block device that uses the lower level block device for the actual storage of information.
To store information on the distributed device, a file system is created on the DRBD logical block device.
@ When used with MySQL, once the file system has been created, you move the MySQL data directory (including InnoDB data files and binary logs) to the new file system.
@
When you set up the secondary DRBD server, you set up the physical block device and the DRBD logical block device that stores the data. The block device data is then copied from the primary to the secondary server.
Installation and configuration sequence
@ First, set up your operating system and environment. This includes setting the correct host name, updating the system and preparing the available packages and software required by DRBD, and configuring a physical block device to be used with the DRBD block device.
@ Installing DRBD requires installing or compiling the DRBD source code and then configuring the DRBD service to set up the block devices to be shared.
@ After configuring DRBD, alter the configuration and storage location of the MySQL data.
@ Optionally, configure high availability using the Linux Heartbeat service.
Setting Up Your Operating System for DRBD
To set your Linux environment for using DRBD, follow these system configuration steps:
@ Make sure that the primary and secondary DRBD servers have the correct host name, and that the host names are unique. You can verify this by using the uname command:
# hostname drbd1 -----> set the hostname for first node
# hostname drbd2 -----> set the hostname for first node
@ Each DRBD node must have a unique IP address. Make sure that the IP address information is set correctly within the network configuration and that the host name and IP address has been set correctly within the /etc/hosts file.
# vim /etc/hosts
192.168.1.231 drbd1 drbd1
# vim /etc/hosts
192.168.1.237 drbd2 drbd2
@ Because the block device data is exchanged over the network,everything that is written to the local disk on the DRBD primary is also written to the network for distribution to the DRBD secondary.
@ You devote a spare disk, or a partition on an existing disk, as the physical storage location for the DRBD data that is replicated. If the disk is unpartitioned, partition the disk using fdisk, cfdisk or other partitioning solution. Do not create a file system on the new partition. (ie, do not partation the new device attached or new partation created).
# fdisk /dev/sdb -----> in primary node create a partation first
n / p(1)
w
# partprobe
# fdisk -l
/dev/sdb1
# fdisk /dev/hdb ----------> create a partation in secondary node also
n / p(1)
w
# partprobe
# fdisk -l
/dev/hdb1
create a new partation OR if you are using a vmware or a virtual box, and you do not have an extra space for a new partation please add an extra data block to have more space. and don't partation the disk. After attaching the drbd device only we need to partation the device. use identical sizes for the partitions on each node, for primary and secondary.
@ If possible, upgrade your system to the latest available Linux kernel for your distribution. Once the kernel has been installed, you must reboot to make the kernel active. To use DRBD, you must also install the relevant kernel development and header files that are required for building kernel modules.
Before you compile or install DRBD, make sure the following tools and files are there
update and install the latest kernel and kernel header files:-
@ root-shell> up2date kernel-smp-devel kernel-smp
@ root-shell> up2date glib-devel openssl-devel libgcrypt-devel glib2-devel pkgconfig ncurses-devel rpm-build rpm-devel redhat-rpm-config gcc gcc-c++ bison flex gnutls-devel lm_sensors-devel net-snmp-devel python-devel bzip2-devel libselinux-devel perl-DBI
# yum install drbd kmod-drbd / if any dependency error came
OR
# yum install drbd82 kmod-drbd82
[/etc/drbd.conf] is the configuration file
To set up a DRBD primary node, configure the DRBD service, create the first DRBD block device, and then create a file system on the device so that you can store files and data.
@ Set the synchronization rate between the two nodes. This is the rate at which devices are synchronized in the background after a disk failure, device replacement or during the initial setup. Keep this in check compared to the speed of your network connection.
@ To set the synchronization rate, edit the rate setting within the syncer block:
Creating your primary node
# vim /etc/drbd.conf
global { usage-count yes; }
common {
syncer {
rate 50M;
verify-alg sha1;
}
handlers { outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater";}
}
resource mysqlha {
protocol C; # Specifies the level of consistency to be used when information
is written to the block device.Data is considered written
when the data has reached the local disk and the
remote node's physical disk.
disk {
on-io-error detach;
fencing resource-only;
#disk-barrier no;
#disk-flushes no;
}
@ Set up some basic authentication. DRBD supports a simple password hash exchange mechanism. This helps to ensure that only those hosts with the same shared secret are able to join the DRBD node group.
net {
cram-hmac-alg sha1;
shared-secret "cEnToS";
sndbuf-size 512k;
max-buffers 8000;
max-epoch-size 8000;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
data-integrity-alg sha1;
}
@ Now you must configure the host information. You must have the node information for the primary and secondary nodes in the drbd.conf file on each host. Configure the following information for each node:
@ device: The path of the logical block device that is created by DRBD.
@ disk: The block device that stores the data.
@ address: The IP address and port number of the host that holds this DRBD device.
@ meta-disk: The location where the metadata about the DRBD device is stored. If you set this to internal, DRBD uses the physical block device to store the information, by recording the metadata within the last sections of the disk.
The exact size depends on the size of the logical block device you have created, but it may involve up to 128MB.
@ The IP address of each on block must match the IP address of the corresponding host. Do not set this value to the IP address of the corresponding primary or secondary in each case.
on drbd1 {
device /dev/drbd0;
#disk /dev/sda3;
disk /dev/sdb1;
address 192.168.1.231:7789;
meta-disk internal;
}
on drbd2 {
device /dev/drbd0;
#disk /dev/sda3;
disk /dev/hdb;
address 192.168.1.237:7789;
meta-disk internal;
}
And in second machine do the same as in first machine
Setting Up a DRBD Secondary Node
The configuration process for setting up a secondary node is the same as for the primary node, except that you do not have to create the file system on the secondary node device, as this information is automatically transferred from the primary node.
@ To set up a secondary node:
Copy the /etc/drbd.conf file from your primary node to your secondary node. It should already contain all the information and configuration that you need, since you had to specify the secondary node IP address and other information for the primary node configuration.
global { usage-count yes; }
common {
syncer {
rate 50M;
verify-alg sha1;
}
handlers { outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater";}
}
resource mysqlha {
protocol C;
disk {
on-io-error detach;
fencing resource-only;
#disk-barrier no;
#disk-flushes no;
}
net {
cram-hmac-alg sha1;
shared-secret "cEnToS";
sndbuf-size 512k;
max-buffers 8000;
max-epoch-size 8000;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
data-integrity-alg sha1;
}
on drbd1 {
device /dev/drbd0;
#disk /dev/sda3;
disk /dev/sdb1;
address 192.168.1.231:7789;
meta-disk internal;
}
on drbd2 {
device /dev/drbd0;
#disk /dev/sda3;
disk /dev/hdb1;
address 192.168.1.237:7789;
meta-disk internal;
}
@@ On both machines, Before starting the primary node and secondary nodes, create the metadata for the devices
# drbdadm create-md mysqlha
@@ On primary/active node,
# /etc/init.d/drbd start ## DRBD should now start and initialize, creating the DRBD devices that you have configured.
DRBD creates a standard block device - to make it usable, you must create a file system on the block device just as you would with any standard disk partition. Before you can create the file system, you must mark the new device as the primary device (that is, where the data is written and stored), and initialize the device. Because this is a destructive operation, you must specify the command line option to overwrite the raw data.
# drbdadm -- --overwrite-data-of-peer primary mysqlha
@ On seconday/passive node,
# /etc/init.d/drbd start
@ On both machines,
# /etc/init.d/drbd status
# cat /proc/drbd ## Monitoring a DRBD Device
• cs: connection state
• st: node state (local/remote)
• ld: local data consistency
• ds: data consistency
• ns: network send
• nr: network receive
• dw: disk write
• dr: disk read
• pe: pending (waiting for ack)
• ua: unack'd (still need to send ack)
• al: access log write count
# watch -n 10 'cat /proc/drbd'
@ On primary/active node,
# mkfs.ext3 /dev/drbd0
# mkdir /drbd ## no needed this bcz we need to mount it in
another point called /usr/local/mysql/
# mount /dev/drbd0 /drbd ## not need this as well
Your primary node is now ready to use.
@ On seconday/passive node,
# mkdir /drbd ## not needed, it will replicate from primary
[[[[[[[[[[ for TESTING the replication of drbd alone follow the above steps. after the primary node is mounted to a mount point, create any files in it. create same mount point in both the system.
# cd /mountpoint
# dd if=/dev/zero of=check bs=1024 count=1000000
After that in primary
# umount /drbd
# drbdadm secondary mysqlfo ## make the primary node as secondary
And in secondary
# drbdadm primary mysqlfo ## make the secondary node as primary
# mount /dev/drbd0 /drbd/
# ls /drbd/ ## the data will be replicated into it. ]]]]]]]]]]]]]]]]]]]]]]]]]]]]]
Mysql for DRBD
# [MySQL] ## install mysql if its not there, here its installed with partation enabled
@ On primary/active node,
# cd mysql-5.5.12/
# cmake . -LH
# cmake .
# make
# make install
# cd /usr/local/mysql/
# chown mysql:mysql . -R
# scripts/mysql_install_db --datadir=/usr/local/mysql/data/ --user=mysql
# scp /etc/my.cnf root@192.168.1.231:/usr/local/mysql/
## config file copied from another machine
# cd /usr/local/mysql/
# vim my.cnf
datadir=/usr/local/mysql/data/
socket=/usr/local/mysql/data/mysql.sock
log-error=/var/log/mysqld.log
pid-file=/usr/local/mysql/mysqld.pid
./bin/mysqld_safe --defaults-file=/usr/local/mysql/my.cnf &
## start mysql server
OR
# nohup sh /usr/local/mysql/bin/mysqld_safe --defaults-file=/usr/local/mysql/my.cnf &
#./bin/mysqladmin -h localhost -uroot password 'mysql'
# vim /etc/profile
export PATH=$PATH:/usr/local/mysql/bin
# . /etc/profile
# mysql -uroot -pmysql
# cd /usr/local/mysql/support-files/
# cp mysql.server /etc/init.d/mysqld
# /etc/init.d/mysqld restart
# /etc/init.d/mysqld stop
### /etc/init.d/drbd stop ---- dont stop drbd
# mkdir /tmp/new
# mv /usr/local/mysql/* /tmp/new/ ## Move the mysql data to a secure location, and mount the drbd partation to /usr/local/mysql
# umount /drbd ## Already mounted partation, need to unmount it and mount the drbd partation to /usr/local/mysql, where mysql data are stored.
# mount /dev/drbd0 /usr/local/mysql/ ## To this location we want to mount drbd where mysql directory locations and installation files resides.
# cp -r /tmp/new/* . ## after mounting the drbd partation to /usr/local/src/ copy the mysql data's back to /usr/local/mysql/ from the alrady backeded place. now the mysql is in drbd partation.
[[[[[[[[[[[[[ for TESTING the mysql replication in drdb
In Primary node
# mysql -uroot -pmysql
mysql> create database DRBD; ## if database is not created the entire mysql instance is replicated to secondary.
# /etc/init.d/mysqld stop ## we stopped this because if the mysql from primary node is stopped we want to replicated the mysql service and db in secondary server.
# umount /usr/local/mysql/ ## umount the mount point in primary
# ls /usr/local/mysql/
# drbdadm secondary mysqlfo ## make primary as secondary node
# /etc/init.d/drbd status
In secondary Node
# drbdadm primary mysqlfo
# /etc/init.d/drbd status
# mount /dev/drbd0 /usr/local/mysql/
# ls /usr/local/mysql/
# /usr/local/mysql/bin/mysql -uroot -pmysql ## we can see the database created in primary replicated to secondary.
# /etcinit.d/mysqld start
]]]]]]]]]]]]]]]]]]]]]]]
Configuring Heartbeat for BRBD (the service attached to DRBD) failover
1. Assign hostname node01 to primary node with IP address 172.16.4.80 to eth0
2. Assign hostname node02 to slave node with IP address 172.16.4.81
Note: on node01
uname -n ---- must return node01
uname -n ---- must return node02
We set the host name already to configure the drbd. Here we use 192.168.1.245 as virtual ip, communications will listen to that IP.
# yum install heartbeat heartbeat-devel ## On both the servers
@ if config files are not under /usr/share/doc/heartbeat
# cd /etc/ha.d/
# touch authkeys
# touch ha.cf
# touch haresources
# vim authkeys
auth 2
2 sha1 test-ha
## auth 3
3 md5 "secret"
# chmod 600 /etc/ha.d/authkeys
# vim ha.cf
logfacility local0
debugfile /var/log/ha-debug
logfile /var/log/ha-log
keepalive 500ms
deadtime 10
warntime 5
initdead 30
mcast eth0 225.0.0.1 694 2 0
ping 192.168.1.22
respawn hacluster /usr/lib/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster
respawn hacluster /usr/lib/heartbeat/dopd
apiauth dopd gid=haclient uid=hacluster
auto_failback off (on)
node drbd1
node drbd2
# vim haresource ## This file contains the information about resources which we want to highly enable.
drbd1 drbddisk Filesystem::/dev/drbd0::/usr/local/mysql::ext3 mysqld 192.168.1.245 (virtual IP)
# cd /etc/ha.d/resource.d
# vim drbddisk
DEFAULTFILE="/etc/drbd.conf"
@ On PRIMARY Node
# cp /etc/rc.d/init.d/mysqld /etc/ha.d/resource.d/
@ Copy the files from primary node to secondary node
# scp -r ha.d root@192.168.1.237:/etc/ ## copy all files to node two, because Primary node and secondary node contains the same configuration.
@@ Stop all services in both the nodes
node1$ service mysqld stop
node1$ umount /usr/local/mysql/
node1$ service drbd stop
node1$ service heartbeat stop
node2$ service mysqld stop
node2$ umount /usr/local/mysql/
node2$ service drbd stop
node2$ service heartbeat stop
@@ # Automatic startup,
node1$ chkconfig drbd on
node2$ chkconfig drbd on
node1$ chkconfig mysqld off ## mysql will be handled by heartbeat, its exe we placed in /ha.d/resources/
node2$ chkconfig mysqld off
node1$ chkconfig heartbeat on
node2$ chkconfig heartbeat on
# Start drbd on both machines,
node1$ service drbd start
node1$ service heartbeat start
# Start heartbeat on both machines,
node2$ service drbd start
node2$ service heartbeat start
No need of starting mysql Heartbeat will start it automatically.
For testing the replication
#/usr/lib/heartbeat/hb_standby ## Run this command in any host, then that host will going down and the DB will replicate to other system
@ access the DB from a remote host using the virtual IP
mysql> grant all privileges on *.* to 'root'@'192.168.1.67' identified by 'mysql1';
mysql> flush privileges;
# delete from user where Host='192.168.1.67'
# mysql -uroot -p -h 192.168.1.245
#[Test Failover Services]
node1$ hb_standby
node2$ hb_takeover
#[Sanity Checks]
node1$ service heartbeat stop
node2$ service heartbeat stop
$/usr/lib64/heartbeat/BasicSanityCheck
#[commands]
$/usr/lib64/heartbeat/heartbeat -s
