Monday 9 December 2013

1. Networking Fundamentals Master the OSI Model and TCP IP


OSI Model:-  Basic standards for network communication

Protocols 


Standards of communication : Packaging / Addressing / Payment 

/ Getting the package on the network
   
Layers: Application
        Presentation
        Session
        Transport
        Network
        Data Link
        Physical

Here remember layers numbered from bottom to top so Layer 3 means Network layer in OSI model.

 Function of each Layer


Application:- Deals with Network API's(Application programming interface's). It is an interface between your application and Operating Sytem. (Interfaces which communicates with your applications)

Presentation:- Deals with the formatting of the information that going out on the network.

Session:- synchronization / sending and receiving computer in synch with one another.

Transport:- Packet management, Data when it is sent, it is broken into packets and transport layer which manages those packets.
Keeping track of how many they are and did they get their.
              
Network:- Deals with addressing and routing, addressing with TCP/IP, And routing from one network to another.

Data:- Deals with data frames, frames means ethernet frames, Tocken ring frames etc. What kind of network we are using to package this datas.
               
Physical:- All about the Hardware, Network cards attached to the cables. an interface between application and OSI model.

 
  Sending Computer                  Receiving Computer

    Application     --------------     Application
       |                                  ^
       |                                  |
       >                                  |
    Presentation --------------        Presentation
       |                                  ^
       |                                  |
       >                                  |
    Session        --------------      Session
       |                                  ^
       |                                  |
       >                                  |
    Transport    --------------         Transport   
       |                                  ^
       |                                  |
       >                                  |
    Network        --------------       Network   
       |                                  ^
       |                                  |
       >                                  |
     Data        --------------         Data
       |                                  ^
       |                                  |
       >                                  |
    Physical    --------------         Physical   
       |                                  ^
       |                                  |
       |__________________________________|     
                   Network Cable
     

Arrows goes down through on sending side  And they go up through the OSI model on the Receiving side. that is important,

When sending some data, on sending side it starts from Application layer to Physical layer, and pick's up network cables and on Receiving end it goes up starting from Physical layer to Application Layer and each layer acknowledges what the corresponding layer on sending side added to the information.

When data goes down from application layer to physical layer while sending some data's will append to front or back ie, header or trailer of the data.  and it travels through Network cable and reaches to the Physical layer of the Receiving end and moves up to Application layer and each layer tripping off  the data of corresponding layer from the sending side has put on.

Device's in each Layer


Phyical Layer: USB, HUB, Bluetooth, NIC card

Data Link Layer: Switch(do filtering based on computers mac address)
Network: Router, IPV4, IPV6 and ICMP

 Rest three layers has no hardwares in it only protocols
 Transport: TCP, UDP
 Session: PPTP, TLS/SSL
 Presentation: MIME

Layer 3 lavel VPN Router:- 
         Network Layer, bcz layer counting from bottom to top.
       
 
 TCP/IP Model

Application: it will do the functionality of presentation and session layer of OSI model
Transport:
Internet:
Network Access:

 TCP/IP Protocols


Application:- HTTP, FTP, SNMP, DNS, SMTP
Transport:- TCP,UPD
Internet:- IP, ICMP,
Network Access:- Ethernet, Token Ring

TCP, Connection Oriented:- Connection Must be Established before data exchange, It will establish with something called Three-way Handshake
  
UDP, Connection less:- and un-reliable. Many used for Multi cast addressing ie, One computer is broadcasting and many listeners.

ICMP is used for Diagnostic and error reporting. icmp is used with UDP # ping

Protocol is a set of rules and procedure for communication.

 ThreeWay HandShake

 1.  The client sends a SYN packet to the server indicating
that it wants to set a TCP connection.It also sends ISN (Initial Sequenc Number). Here ISN is x.

2. If the server is 'alive' and listening on the requested
 port and can accept an incoming connection, it replies with its own SYN + ACK packet. It sends its own ISN (Initial Sequence Number)(for this connection, y ) and acknowledges the clients request by sending back  client's ISN + 1 sequence number (x + 1).

3. Finally, after receiving the server's SYN + ACK response, the client sends back an ACK packet with a sequence number of server's ISN + 1 (y + 1).

 In Some Easy Steps:-

  Host A sends a TCP SYNchronize packet to Host B
  Host B receives A's SYN
  Host B sends a SYNchronize-ACKnowledgement
  Host A receives B's SYN-ACK
  Host A sends ACKnowledge
  Host B receives ACK.
  TCP socket connection is ESTABLISHED.   
 

cut | rev | wc | sort | uniq | diff | tr | ls | find


Cut

Cut out selected fields of each line of a file

Extracting text by column:
# cut -f3 -d: /etc/passwd (display third colon(:)-delimited field)
# cut -c5 /etc/passwd    (display 5th character)
# cut -c1-5 /etc/passwd  (display first 5 characters)

-d  specify the column delimiter (default is TAB)
-f   specify column to print
-c   cut by character

1.Display the 1st field (employee name) from a colon delimited file
# cut -f1 -d" "  names.txt   ---> Here delimiter is space

2.Display 1st and 3rd field from a colon delimited file
# cut -f1,3 -d:  names.txt
# cut -d: -f1,3 /etc/passwd

Emma Thomas:Marketing
Alex Jason:Sales
Madison Randy:Product Development
Sanjay Gupta:Support
Nisha Singh:Sales

3.Display only the first 8 characters of every line in a file
# cut -c1-8 /etc/passwd

oprofile
rpcuser:
nfsnobod
xfs:x:43
haldaemo
avahi-au
gdm:x:42
sabayon:
vboxadd:

4. To print the characters from tenth position to the end, specify only the start position and omit the end position.
# cut -c10- file.txt
 

rev  --- reverse

Reverse lines of a file



First reverse the text in each line and then apply the command on it.
# rev filenames.txt | cut -d'.' -f1


Word Count (wc)
 
The wc (word count) command in Unix/Linux operating systems is used to find out number ofnewline count, word count, byte and characters count in a files specified by the file arguments.


wc -l : Prints the number of lines in a file.
wc -w : prints the number of words in a file.
wc -c : Displays the count of bytes in a file.
wc -m : prints the count of characters from a file.
wc -L : prints only the length of the longest line in a file.


# wc file1              
(displays no. of lines, words and character in file1)

# cat names2.txt
wali
salman
obama
wali
wali
wali
ajay
sameer

# wc  *                 
(displays no. of lines, words and character of every files in the current directory)

-l only for line count
-w only for word count
-c only for byte count
-m only for character count (1 character = 1 byte)
 

 Sort

Sort command is helpful to sort/order lines in text files. You can sort the data in text file and display the output on the screen, or redirect it to a file.

# grep bash /etc/passwd | sort

(sort the UIDs in ascending order)
# sort  -t:  -k3  -n  /etc/passwd    

(shows only UIDs in ascending order)
# sort  -t:  -k3  -n  /etc/passwd | cut  -f3  -d:

-r performs a reverse (descending) sort
-n performs a numeric sort
-f ignores (folds) case of characters in strings
-u (unique) removes duplicate lines in output
-t: uses : as a filed separator
-k3 third column by : delimited field

  
uniq

Uniq command is helpful to remove or detect duplicate entries in a file. 

1. Eliminating duplicate lines:

cat>file
Emma Thomas:Marketing
Alex Jason:Sales
Madison Randy:Product Development
Sanjay Gupta:Support
Nisha Singh:Sales
wali
salman
obama
wali
wali
wali
ajay
sameer

# uniq file
(uniq without argument, removes duplicate adjacent lines)

Emma Thomas:Marketing
Alex Jason:Sales
Madison Randy:Product Development
Sanjay Gupta:Support
Nisha Singh:Sales
wali
salman
obama
wali
ajay
sameer

-u to output only the lines that are truly unique, only occurring once in the input.


# uniq -u names.txt

Emma Thomas:Marketing
Alex Jason:Sales
Madison Randy:Product Development
Sanjay Gupta:Support
Nisha Singh:Sales
wali
salman
obama
ajay
sameer

-d to output only print one copy of the lines that are repeated in the input, Duplicate lines.


# uniq -d names.txt
wali

-c each line will be prepended with a number indicating how many times it appears in the input.
 

# uniq -c names.txt
      1 Emma Thomas:Marketing
      1 Alex Jason:Sales
      1 Madison Randy:Product Development
      1 Sanjay Gupta:Support
      1 Nisha Singh:Sales
      1 wali
      1 salman
      1 obama
      3 wali
      1 ajay
      1 sameer


Comparing files (diff)

Displays two files and prints the lines that are different.

# diff  file  file1

# diff  -u  file  file1  
(line that begin with + exist in names2.txt but not in names.txt,
line that begin with - exist in names.txt but not in names2.txt)

# diff -u names.txt names2.txt
--- names.txt   2013-11-02 00:15:44.000000000 +0530
+++ names2.txt  2013-11-02 00:25:29.000000000 +0530
@@ -1,8 +1,3 @@
-Emma Thomas:Marketing
-Alex Jason:Sales
-Madison Randy:Product Development
-Sanjay Gupta:Support
-Nisha Singh:Sales
 wali
 salman
 obama


# Colordiff file1 file2

 tr
 
tr is an UNIX utility for translating, or deleting, or squeezing repeated characters. It will read from STDIN and write to STDOUT.

1. Convert lower case to upper case

# tr abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ

# tr a-z A-Z
thegeekstuff
THEGEEKSTUFF

2. Translate braces into parenthesis

You can also translate from and to a file. In this example we will translate braces in a file with parenthesis.

$ tr '{}' '()' < inputfile > outputfile

The above command will read each character from “inputfile”, translate if it is a brace, and write the output in “outputfile”.

3. Translate white-space to tabs

The following command will translate all the white-space to tabs

$ echo "This is for testing" | tr [:space:] '\t'
This    is    for    testing

4. Squeeze repetition of characters using -s

 you can convert multiple continuous spaces with a single space

$ echo "This  is  for testing" | tr -s [:space:] ' '
This is for testing

5. Delete specified characters using -d option

tr can also be used to remove particular characters using -d option.

$ echo "the geek stuff" | tr -d 't'
he geek suff

To remove all the digits from the string, use

$ echo "my username is 432234" | tr -d [:digit:]
my username is

6. Complement the sets using -c option

You can complement the SET1 using -c option. For example, to remove all characters except digits, you can use the following.

$ echo "my username is 432234" | tr -cd [:digit:]
432234

7. Join all the lines in a file into a single line

The below command will translate all newlines into spaces and make the result as a single line.

$ tr -s '\n' ' ' < file.txt


 LS

Listing files and directorie's

1. -t  sort by modification time

2. Display One File Per Line Using # ls -1
    Display All Information About Files/Directories # ls -l

 -rw-r--r-- 1 root root 6066 Oct 28 12:53 install.log.syslog
 drwxr-xr-x 4 root root 4096 Nov 1 12:04 j

1st Character – File Type: First character specifies the type of the file.


In the example above the hyphen (-) in the 1st character indicates that this is a normal file. Following are the possible file type options in the 1st character of the ls -l output.

    Field Explanation
    - normal file
    d directory
    s socket file
    l link file
   
Field 1 – File Permissions
Field 2 – Number of links: Second field specifies the number of links for that file. In this example, 1 indicates only one link to this file.
Field 3 – Owner
Field 4 – Group
Field 5 – Size
Field 6 – Last modified date & time
Field 7 – File name

3. Display File Size in Human Readable Format Using 

    # ls -lh

4. Display Directory Information Using 

   # ls -ld

5. Order Files Based on Last Modified Time Using 

   # ls -lt

6. Order Files Based on Last Modified Time (In Reverse Order) Using

   # ls -ltr

7. Display Hidden Files Using 

   # ls -a (or) # ls -A

It will show all the files including the ‘.’ (current directory) and ‘..’ (parent directory).
 

To show the hidden files, but not the ‘.’ (current directory) and ‘..’ (parent directory), use option -A.
     
8. Display Files Recursively Using # ls -R

# ls -R
.:
anaconda-ks.cfg  Desktop  install.log  install.log.syslog  ipt  j  lvm-sizefile  lvmsnap1.tar

./Desktop:
LVM  LVM~

./j:
32Bit  AUTORUN.INF  VBoxLinuxAdditions-amd64.run  VBoxSolarisAdditions.pkg        VBoxWindowsAdditions.exe
64Bit  autorun.sh   VBoxLinuxAdditions-x86.run    VBoxWindowsAdditions-amd64.exe  VBoxWindowsAdditions-x86.exe

./j/32Bit:
OS2  Readme.txt

./j/32Bit/OS2:
gengradd.dll  libc063.dll  readme.txt  VBoxGuest.sys  vboxmouse.sys  VBoxService.exe

./j/64Bit:
Readme.txt

It will go through all dir in the current location and display the contents.


9. Display File Inode Number/iNode number Using 

  # ls -i

10. Display File UID and GID Using 

   # ls -n

11. Visual Classification of Files With Special Characters Using 

  # ls -F

Find

Find the passwd file under root and two levels down
# find / -maxdepth 3 -name passwd

Min depth and Max depth
# find -mindepth 3 -maxdepth 5 -name passwd

Inverting the match
# find -maxdepth 1 -not -iname "MyCProgram.c"

Find file by inode number
# find -inum 16187430

Find files which has read permission only to group
# find . -perm g=r -type f -exec ls -l {} \;

Find files which has read permission only to group
# find . -perm 040 -type f -exec ls -l {} \;

Find all empty files (zero byte file) in your home directory and its sub-directory# find ~ -empty

List all the empty files only in your home directory.
# find . -maxdepth 1 -empty

List only the non-hidden empty files only in the current directory.
# find . -maxdepth 1 -empty -not -name ".*"

Finding the Top 5 Big Files
# find . -type f -exec ls -s {} \; | sort -n -r | head -5

Finding the Top 5 Small Files. Technique is same as finding the bigger files, but the only difference the sort is ascending order.# find . -type f -exec ls -s {} \; | sort -n  | head -5

---> Find Files Based on file-type using option -type:-

Find only the socket files.
# find . -type s

Find all directories
# find . -type d

Find only the normal files
# find . -type f

Find all the hidden files
# find . -type f -name ".*"

Find all the hidden directories
# find -type d -name ".*"


---> Find Files by Size

Find files bigger than the given size
# find ~ -size +100M

Find files smaller than the given size
# find ~ -size -100M

Find files that matches the exact given size
# find ~ -size 100M

---> Remove big archive files using find command

The following command removes *.zip files that are over 100M.
# find / -type f -name *.zip -size +100M -exec rm -i {} \;"

---> Find files whose content got updated within last 1 hour
 -mmin n File’s data was last modified n minutes ago
 -mtime n File’s data was last modified n*24 hours ago


Find files in the current directory and sub-directories, whose content got updated within last 1 hour (60 minutes) # find . -mmin -60

Finds all the files (under root file system /) that got updated within the last 24 hours (1 day).# find / -mtime -1

---> Find files which got accessed before 1 hour
 -amin n File was last accessed n minutes ago
 -atime n File was last accessed n*24 hours ago


Find files in the current directory and sub-directories, which got accessed within last 1 hour (60 minutes)# find -amin -60

Finds all the files (under root file system /) that got accessed within the last 24 hours (1 day).# find / -atime -1

---> Find files which got changed exactly before 1 hour
 -cmin n File’s status was last changed n minutes ago.
 -ctime n File’s status was last changed n*24 hours ago.

Find files in the current directory and sub-directories, which changed within last 1 hour (60 minutes)# find . -cmin -60

Finds all the files (under root file system /) that got changed within the last 24 hours (1 day).# find / -ctime -1

---> Long list the files which are edited within the last 1 hour.
# find -mmin -60
./cron
./secure

# find -mmin -60 -exec ls -l {} \;
-rw-------  1 root root 1028 Jun 21 15:01 ./cron
-rw-------  1 root root 831752 Jun 21 15:42 ./secure

DoS Attack Detection In Linux


You can simply use netstat command to print out a list of all open connection to your Linux box. The list will be sorted out using sort command including total number of connections from a specific IP address.

Login as the root user and type the following command,

# netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n


Sunday 8 December 2013

How to Install VSFTPD Server

# yum install vsftpd (Very Secure FTP Daemon)

Testing ftp connection

# ps -aux | grep ftp
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
root      9342  0.0  0.0   5332   524 ?        Ss   00:07   0:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf
root      9345  0.0  0.0   4032   684 pts/2    R+   00:07   0:00 grep ftp

[root@localhost ~]# netstat -a | grep ftp
tcp  0   0  *:ftp    *:*    LISTEN

VSFTPD only reads the contents of its vsftpd.conf configuration file only when it starts.

Vsftpd Defaults:-

Default port: TCP/UDP - 21 and 20
The main configuration file: /etc/vsftpd/vsftpd.conf
Users that are not allowed to login via ftp: /etc/vsftpd/ftpusers

Configuration file
# vim /etc/vsftpd/vsftpd.conf

We can enable anonymous user login and local user login into this. anonymous user login will chrooted to /var/ftp and Local user login will chrooted to his/her home directory ie, /home/ctechz/. These by Default.

# Allow anonymous FTP? (Beware - allowed by default if you comment this out).
anonymous_enable=NO

# Uncomment this to allow local users to log in.
local_enable=YES

# Uncomment this to enable any form of FTP write command.
write_enable=YES

# Default umask for local users is 077. You may wish to change this to 022, if your users expect that (022 is used by most other ftpd's)
local_umask=022

# Activate directory messages - messages given to remote users when they go into a certain directory.
dirmessage_enable=YES

# The target log file can be vsftpd_log_file or xferlog_file.
# This depends on setting xferlog_std_format parameter
xferlog_enable=YES

# Make sure PORT transfer connections originate from port 20 (ftp-data).
connect_from_port_20=YES

# The name of log file when xferlog_enable=YES and xferlog_std_format=YES


# WARNING - changing this filename affects /etc/logrotate.d/vsftpd.log
xferlog_file=/var/log/xferlog

# By default the server will pretend to allow ASCII mode but in fact ignore the request. Turn on the below options to have the server actually do ASCII mangling on files when in ASCII mode.
# Beware that on some FTP servers, ASCII support allows a denial of service attack (DoS) via the command "SIZE /big/file" in ASCII mode. vsftpd predicted this attack and has always been safe, reporting the size of the raw file.
# ASCII mangling is a horrible feature of the protocol.
ascii_upload_enable=YES
ascii_download_enable=YES

# You may fully customise the login banner string:
ftpd_banner=Welcome to CtechZ FTP service.

#Finish up by uncommenting command to chroot_local_user.
#When this line is set to Yes, all the local users will be jailed within their chroot and will be denied access to any other part of the server. If we didnt specify this a local user can browse entire system folder's
chroot_local_user=YES

# When "listen" directive is enabled, vsftpd runs in standalone mode and listens on IPv4 sockets. This directive cannot be used in conjunction with the listen_ipv6 directive.
listen=YES

pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES
use_localtime=YES

#To restrict Linux FTP users in their own home directory Path
#If these entries are missing then you can insert in the vsftpd.conf file at bottom
userlist_deny=YES
userlist_file=/etc/vsftpd/ftpusers


# Use the Below Lines only if you need to change the default FTP Ports



# Uncomment the below lines if you need to work ftp in default ports
#To changing the default port
listen_port=2021

# data transfering port, Default port is 20 to change this. / For #Active FTP
#Specifies the port used for active data connections when #connect_from_port_20 is set to YES.
#The default value is 20
ftp_data_port=2022

#Configuring vsftpd for Passive data connection
pasv_min_port=1024
pasv_max_port=1025


 Creating an FTP directory and User

# useradd -s /sbin/nologin -d /CtechZ-FTP/ cftp
 

# passwd cftp
    ftp@123#


 To stop users to do ftp even to home directory

# vim /etc/vsftpd/ftpusers
 and enter the username here,that user can't do ftp into the server.
 

 IP table Rules for ftp active and Passive connections

# Here i am forwarding from a single machine not from a router and its a stateful firewall rule. 

# Allowing INCOMING FTP 21/20 connections
# You'll need an additional rule to allow "related" connections. This is due to the FTP protocol using one port for commands and another for data.

-A INPUT -i eth0 -p tcp  --dport 21 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT -m comment --comment "Allow ftp connections on port 21"

-A OUTPUT -o eth0 -p tcp --sport 21 -m state --state ESTABLISHED,RELATED -j ACCEPT -m comment --comment "Allow ftp connections on port 21"


# Same in Different Ports / Allowing InComing FTP 2021/2021 connections / Make changes in vsftpd.conf file for different port

-A INPUT -i eth0 -p tcp  --dport 2021 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT -m comment --comment "Allow ftp connections on port 21"
 

-A OUTPUT -o eth0 -p tcp --sport 2021 -m state --state ESTABLISHED,RELATED -j ACCEPT -m comment --comment "Allow ftp connections on port 21"

#
Allowing Active FTP
-A OUTPUT -o eth0 -p tcp --sport 20 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT -m comment --comment "Allow ftp connections on port 20"
 

-A INPUT -i eth0 -p tcp --dport 20 -m state --state ESTABLISHED,RELATED -j ACCEPT -m comment --comment "Allow ftp connections on port 20"
 

# Same in Different Ports / Allowing Active FTP 2022/2022 connections / Make changes in vsftpd.conf file for different port 
-A OUTPUT -o eth0 -p tcp --sport 2022 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
 

-A INPUT -i eth0 -p tcp --dport 2022 -m state --state ESTABLISHED,RELATED -j ACCEPT -m comment --comment "Allow ftp connections on port 20"

#
Allowing Passive FTP 
-A OUTPUT -o eth0 -p tcp --sport 1024:1025 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
 

-A INPUT -i eth0 -p tcp --dport 1024:1025 -m state --state ESTABLISHED,RELATED -j ACCEPT -m comment --comment "Allow ftp connections on port 20" 

  Loading Iptable Kernal Modules for FTP

Put the modules you want to load in /etc/rc.local file, as it is the last file reading when machine is loading,

# vim /etc/rc.local

# Module to track the state of connections /#iptables
modprobe ip_conntrack
# Load the iptables active FTP module, requires ip_conntrack /#iptables
modprobe ip_conntrack_ftp
# Load iptables NAT module when required /#iptables
modprobe iptable_nat
# Module required for active an FTP server using NAT /#iptables
modprobe ip_nat_ftp

OR

Put these above modules in /etc/sysconfig/iptables-config file

# vim /etc/sysconfig/iptables-config
 

IPTABLES_MODULES="ip_conntrack_netbios_ns"
 

IPTABLES_MODULES="ip_conntrack_ftp"
 

IPTABLES_MODULES="ip_conntrack"
 

IPTABLES_MODULES="iptable_nat"
 

IPTABLES_MODULES="ip_nat_ftp"

Monday 18 November 2013

Linux Quota Management

Quota System allows an administrator to establish limits on the amount of disk resources user can consume.

The Quota is set in the /home partation. usrquota for user and grpquota for group.

If we want user quota first initialize quota in fstab

# vim /etc/fstab
/home /home ext3 defaults,usrquota 0 0
                         ,grpquota

# mount -o remount /home

# quotacheck -c /home  ------ which create a database file

             -cg /home                aquota.user in /home
            

# edquota -u username
          -g groupname


Start the assigned quota by

# quotaon /home
  quotaoff /home
 
block ----- used space by system
soft ----- get warning
hard ---- Max limit, quota exceeding size
inode ---- number of files


while giving soft and hard limits, add block size also with hard and soft, ie,if block size is 32, when you giving soft(100) and hard limit(200) it looks like 132 and 232

To checck quota first check as that user

# su - ctechz
Then create a file in his home dir with some limits exceeds

dd if=/dev/zero of=anyfileName bs=1024 count=5 (files generated which has that much size)

For checking the status of user quota

# repquota /home

To delete quota

# quotaoff /home
# vim /etc/fstab
/home /home ext3 defaults 0 0  ---- remove usrquota from here

# edquota -u ctechz
    and remove the Hard and Soft
   
# quotaoff /home

# mount -o remount /home

LVM export and import: Move a VG to another machine or group

vgexport & vgimport are not necessary to move drives from one system from another.

It is an admin policy tool to prevent access to volumes in the time it takes to move them.

 Exporting Volume Group

1. unmount the file system

First make sure no users are accessing files on active volume, then unmount it

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm
/dev/mapper/vg--ctechz-lvm--ctechz
                      664M  542M   90M  86% /lvm-ctechz

# umount /lvm-ctechz/

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm

2. Mark the Volume Group inactive

Marks the volume group inactive removes it from the kernal and prevents any further activity on it.

# vgchange -an vg-ctechz(VG name)
0 logical volume(s) in volume group "vg-ctechz" now active

3. Export the VG

It is now necessor to export the Volume Group, this prevents it from being accessed on the "old"host system and prepares it to be removed.

# vgexport vg-ctechz(vg name)
  Volume group "vg-ctechz" successfully exported

when the machine is next shut down, the disk can be unplgged and then connected to its new machine.

 Import the Volume Group(VG)

When plugged into new system it becomes /dev/sdb or what ever depends so an initial pvscan shows:

1. # pvscan
PV /dev/sda3 is in exported VG vg-ctechz[580.00MB/0 free]
PV /dev/sda4 is in exported VG vg-ctechz[484.00MB/312.00MB free]
PV /dev/sda5 is in exported VG vg-ctechz[288.00MB/288.00MB free]
 Total: 3 [1.32 GB] / in use: 3 [1.32 GB] / in no VG: 0[0]

2. We can now import the Volume Group (which also activates it) and mount the fle system.

If you are importing on an LVM2 system run,

# vgimport vg-ctechz
Volume group "vg-ctechz" successfully imported

If you are importing on an LVM1 system, add the pvs that needed to import

# vgimport vg-ctechz /dev/sda3 /dev/sda4 /dev/sda5

3. Activate the Volume Group

You must activate the volume group before you can access it

# vgchange -ay vg-ctechz
1 logical volume(s) in volume group "vg-ctechz" now active

Now mount the file system
# mount /dev/vg-ctechz/lvm-ctechz /LVM-import/

# mount
/dev/sda1 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/mapper/vg--ctechz-lvm--ctechz on /LVM-import type ext3 (rw)
[root@localhost ~]#

[root@localhost ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm
/dev/mapper/vg--ctechz-lvm--ctechz
                      664M  542M   90M  86% /LVM-import

 Using Vgscan

# pvs

  PV         VG        Fmt  Attr PSize   PFree
  /dev/sda3  vg-ctechz lvm2 ax-  580.00M 0
  /dev/sda4  vg-ctechz lvm2 ax-  484.00M 312.00M
  /dev/sda5  vg-ctechz lvm2 ax-  288.00M 288.00M

# pvs shows in which all disk attached to vg

# vgscan
Reading all physical volumes.  This may take a while...
Found exported volume group "vg-ctechz" using metadata type lvm2

# vgimport vg-ctechz
Volume group "vg-ctechz" successfully imported

# vgchange -ay vg-ctechz
1 logical volume(s) in volume group "vg-ctechz" now active

# mkdir /LVM-vgscan
# mount /dev/vg-ctechz/lvm-ctechz /LVM-vgscan

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm
/dev/mapper/vg--ctechz-lvm--ctechz
                      664M  542M   90M  86% /LVM-vgscan

# mount
/dev/sda1 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/mapper/vg--ctechz-lvm--ctechz on /LVM-vgscan type ext3 (rw)

VG Scan is using when we are not exporting the vg. ie, first umount the Logical Volume and take the disk and attach it to some other disk, and then do the # vgscan

it will detect the volume group from the disk and mount it in the new system.

How to remove LVM Snapshot

When backup has finished you can now unmount the volume and remove it from the system. you should remove snapshot volume when you have finished with them because then take a copy of all data written to the original volume and this can hurt performance,

# vgdisplay
  --- Volume group ---
  VG Name               vg-ctechz
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               1.04 GB
  PE Size               4.00 MB
  Total PE              266
 Alloc PE / Size       250 / 1000.00 MB
  Free  PE / Size       16 / 64.00 MB
  VG UUID         ui4JTr-JOwC-VCvG-5Evc-poOD-3Klr-hM3feq

# vgs
  VG        #PV #LV #SN Attr   VSize VFree
  vg-ctechz   2   2   1 wz--n- 1.04G 64.00M


# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm
/dev/mapper/vg--ctechz-lvm--ctechz
                      591M  542M   20M  97% /lvm-ctechz
/dev/mapper/vg--ctechz-lvm--ctechz--spapshot1
                      591M  542M   20M  97% /snapshot

# umount /snapshot/

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm
/dev/mapper/vg--ctechz-lvm--ctechz
                      591M  542M   20M  97% /lvm-ctechz

# ls /dev/vg-ctechz/
lvm-ctechz  lvm-ctechz-spapshot1

# lvremove /dev/vg-ctechz/lvm-ctechz-spapshot1
Do you really want to remove active logical volume lvm-ctechz-spapshot1? [y/n]: y
Logical volume "lvm-ctechz-spapshot1" successfully removed

# vgs
  VG        #PV #LV #SN Attr   VSize VFree
  vg-ctechz   2   1   0 wz--n- 1.04G 464.00M

We got our 400MB back in Volume Group


How to take LMV Snapshots / Backuping LVM via snapshot

A snapshot volume is a special type of volume that presents all the data that was in the volume at the time the snapshot was created.

This means we can back up that volume without having to worry about data being changed while the backup is going on,and we don't have to take the database volume offline while the backup is taking place.

snapshot is just a link which point to another place, if the file is growing in lvm and we need a backup of data at a particular time say @ 6:00 pm, First make a snapshot at 6:00 and and mount it some where and take backup of data from that place. This allows the administrator to create a new block device which presents an exact copy of a logical volume, frozen at some point in time. 


Typically this would be used when some batch processing, a backup for instance, needs to be performed on the logical volume, but you don't want to halt a live system that is changing the data.


When the snapshot device has been finished with the system administrator can just remove the device.


This facility does require that the snapshot be made at a time when the data on the logical volume is in a consistent state.

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm
/dev/mapper/vg--ctechz-lvm--ctechz
                      591M  542M   20M  97% /lvm-ctechz


While we are creating a lvm snapshot we are taking space from volume group, so first make sure volume group contain enough free space.

# vgs or vgdisplay

Make sure the snapshot has enough free space or same as origin size(or you can create it in any size)

1.Check in Volume group for free space
# vgs

# vgs
VG        #PV #LV #SN Attr   VSize VFree
vg-ctechz   2   2   1 wz--n- 1.04G 464.00M

free space is available so create a snapshot of current lvm using the vg

# lvcreate -L +400M -s /dev/vg-ctechz/lvm-ctechz -n lvm-ctechz-spapshot1
Logical volume "lvm-ctechz-spapshot1" created


here in plaec of +400M we can give any size, we can even mention +10M as well
-L|--size LogicalVolumeSize
-s|--snapshot} OriginalLogicalVolume[Path]
-n|--name LogicalVolumeName


lvcreate -L +size -s snapshotName
-n lvmName

# ls /dev/vg-ctechz/
lvm-ctechz  lvm-ctechz-spapshot1

# mkdir /snapshot
# mount /dev/vg-ctechz/lvm-ctechz-spapshot1 /snapshot/

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm
/dev/mapper/vg--ctechz-lvm--ctechz
                      591M  542M   20M  97% /lvm-ctechz
/dev/mapper/vg--ctechz-lvm--ctechz--spapshot1
                      591M  542M   20M  97% /snapshot
 
# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg-ctechz/lvm-ctechz
  VG Name                vg-ctechz
  LV UUID           jZEuoN-16MG-30eX-SZaI-8ETO-ZguH-YRVyeN
  LV Write Access        read/write
  LV snapshot status    source of
              /dev/vg-ctechz/lvm-ctechz-spapshot1[active]
  LV Status              available
   # open                 1
  LV Size                600.00 MB
  Current LE             150
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

  --- Logical volume ---
  LV Name                /dev/vg-ctechz/lvm-ctechz-spapshot1
  VG Name                vg-ctechz
  LV UUID                kk3Cvm-Il4z-J8Kt-qt8s-TstZ-Peyw-gU0Uho
  LV Write Access        read/write
  LV snapshot status     active destination for 

                         /dev/vg-ctech/lvm-ctechz
  LV Status              available
  # open                 1
  LV Size                600.00 MB
  Current LE             150
  COW-table size         400.00 MB
  COW-table LE           100
  Allocated to snapshot  0.01%
  Snapshot chunk size    4.00 KB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1


# lvdisplay /dev/vg-ctechz/lvm-ctechz
--- Logical volume ---
  LV Name                /dev/vg-ctechz/lvm-ctechz
  VG Name                vg-ctechz
  LV UUID                jZEuoN-16MG-30eX-SZaI-8ETO-ZguH-YRVyeN
  LV Write Access        read/write
LV snapshot status     source of 

               /dev/vg-ctechz/lvm-ctechz-spapshot1[active]
  LV Status              available
  # open                 1
  LV Size                600.00 MB
  Current LE             150
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

Here we can get the status of the active snapshot

Now we created the Snapshot for the Logical Volume what we have and then do the backup, so first create the snapshot and then backup. 


For backuping use your methods or strategy 

# tar -cvf lvmsnap1.tar /dev/mapper/vg--ctechz-lvm--ctechz--spapshot1

How to Extend Volume Group / Adding physical volumes to a volume group

For increasing the volume group size create another partation in the same disk or in a separate disk,

# vgdisplay
  --- Volume group ---
  VG Name               vg-ctechz
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  16
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               1.04 GB
  PE Size               4.00 MB
  Total PE              266
  Alloc PE / Size       188 / 752.00 MB
  Free  PE / Size       78 / 312.00 MB
  VG UUID       ui4JTr-JOwC-VCvG-5Evc-poOD-3Klr-hM3feq

[root@localhost ~]# vgs
  VG        #PV #LV #SN Attr  VSize VFree
  vg-ctechz  2   1   0 wz--n- 1.04G 312.00M

# fdisk -l

Disk /dev/sda: 31.8 GB, 31890341888 bytes
255 heads, 63 sectors/track, 3877 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot  Start End  Blocks   Id  System
/dev/sda1   *    1    3315 26627706 83  Linux
/dev/sda2       3316  3446 1052257+ 82  Linux swap/Solaris
/dev/sda3       3447  3520 594405   8e  Linux LVM
/dev/sda4       3521  3582 498015   8e  Linux LVM

# fdisk /dev/sda

The number of cylinders for this disk is set to 3877.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
e
Selected partition 2
First cylinder (3316-3877, default 3316):
Using default value 3316
Last cylinder or +size or +sizeM or +sizeK (3316-3446, default 3446): +600M

Command (m for help): p

Disk /dev/sda: 31.8 GB, 31890341888 bytes
255 heads, 63 sectors/track, 3877 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot  Start  End Blocks   Id  System
/dev/sda1   *    1     3315 26627706 83  Linux
/dev/sda2       3316   3389 594405   5   Extended
/dev/sda3       3447   3520 594405   8e  Linux LVM
/dev/sda4       3521   3582 498015   8e  Linux LVM

Command (m for help):
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

# partprobe

then create the new partition inside the extended partition,

# fdisk /dev/sda2

# fdisk -l

Disk /dev/sda: 31.8 GB, 31890341888 bytes
255 heads, 63 sectors/track, 3877 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot Start  End   Blocks   Id  System
/dev/sda1   *  1      3315  26627706 83  Linux
/dev/sda2      3316   3389  594405   5  Extended
/dev/sda3      3447   3520  594405   8e  Linux LVM
/dev/sda4      3521   3582  498015   8e  Linux LVM
/dev/sda5      3316   3352  297171   83  Linux

sda5 is the new partation,

# pvcreate /dev/sda5
  Writing physical volume data to disk "/dev/sda5"
  Physical volume "/dev/sda5" successfully created

# vgs
  VG        #PV #LV #SN Attr   VSize VFree
  vg-ctechz   2   1   0 wz--n- 1.04G 312.00M

to extend vg # vgextend oldvgname /dev/newvgname

# vgextend vg-ctechz /dev/sda5
  Volume group "vg-ctechz" successfully extended

# vgs
  VG        #PV #LV #SN Attr   VSize VFree
  vg-ctechz   3   1   0 wz--n- 1.32G 600.00M

Check the old vgsize and new vgsize here its extended 

from 1.04G to 1.32G

How to reduce LVM

Be careful while reducing LVM, or else their may be a chance of file system get corrupt.

Do this simple technique when you are reducing LVM, leave 10% free when you are reducing say if you are reducing 100GB to 60GB reduce, and when you are reszing leave 10% of 60 = 6 and now 60-6=54. First resize to 54M then reduce to 60M. Check another example,

total space say 300gb and we need the final size after resize to be 200gb then replace 200gb 's 90%  ie, 10*200 / 100 = 20 200-20 = 180Gb. ie, use the 10% ie, use this 180GB when resize the volume and use the correct size(200GB) that we need when reducing the the lvm,

# umount mount point

Check the file system
# e2fsck -f /dev/vgname/lvname

here lvm size is 900MB and need to reduce it to 750MB, then find the 10% of 750Mb ie, 10*750 / 100 = 75 then 750-75=675. so re-size using 675


# resize2fs /dev/vgname/lvname 675M

# lvreduce -L 750M /dev/vgname/lvname

# mount mountPoint

------

# umount /lvm-ctechz

# e2fsck -f /dev/vg-ctechz/lvm-ctechz

e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/vg-ctechz/lvm-ctechz: 12/122880 files (8.3% non-contiguous), 142573/230400 blocks

# resize2fs /dev/vg-ctechz/lvm-ctechz 675M

resize2fs 1.39 (29-May-2006)
Resizing the filesystem on /dev/vg-ctechz/lvm-ctechz to 172800 (4k) blocks.
The filesystem on /dev/vg-ctechz/lvm-ctechz is now 172800 blocks long.

# lvreduce -L 750M /dev/vg-ctechz/lvm-ctechz

  Rounding up size to full physical extent 752.00 MB
  WARNING: Reducing active logical volume to 752.00 MB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce lvm-ctechz? [y/n]: y
  Reducing logical volume lvm-ctechz to 752.00 MB
  Logical volume lvm-ctechz successfully resized

# mount /dev/vg-ctechz/lvm-ctechz /lvm-ctechz/

# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg-ctechz/lvm-ctechz
  VG Name                vg-ctechz
  LV UUID                jZEuoN-16MG-30eX-SZaI-8ETO-ZguH-YRVyeN
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                752.00 MB
  Current LE             188
  Segments               2
  Allocation             inherit
   Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

[root@localhost ~]# lvs
  LV         VG        Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  lvm-ctechz vg-ctechz -wi-ao 752.00M
 

So we successfully reduce our lvm of 900MB to 750MB.

How to extend LVM

say the disk is 97% fill, then what approach you will take

1. check the disk is full or not using df -h

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm
/dev/mapper/vg--ctechz-lvm--ctechz
                      591M  542M   20M  97% /lvm-ctechz


2. Find out the Volume group(VG) containing the logical volume(LV)

# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg-ctechz/lvm-ctechz
  VG Name                vg-ctechz
  LV UUID          jZEuoN-16MG-30eX-SZaI-8ETO-ZguH-YRVyeN
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                600.00 MB
  Current LE             150
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

[root@localhost ~]# lvs
  LV         VG        Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  lvm-ctechz vg-ctechz -wi-ao 600.00M

3. Check the Volume Group(VG) whether we can extend it or not, check is their any free space in volume group or not

# vgdisplay
  --- Volume group ---
  VG Name               vg-ctechz
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  14
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               1.04 GB
  PE Size               4.00 MB
  Total PE              266
  Alloc PE / Size       150 / 600.00 MB
  Free  PE / Size       116 / 464.00 MB
  VG UUID          ui4JTr-JOwC-VCvG-5Evc-poOD-3Klr-hM3feq

[root@localhost ~]# vgs
  VG        #PV #LV #SN Attr   VSize VFree
  vg-ctechz   2   1   0 wz--n- 1.04G 464.00M

If we have free space in VG extend it

# lvextend -L +wanted size/full new size /dev/vgname/lvname

If we want to extend an LVM of size 110MB to 150MB then we can give

# lvextend -L +40M /dev/vgname/lvname  OR

# lvextend -L 150M /dev/vgname/lvname

We can also give it with block size

# lvextend -l +multiple of PE size /dev/vgname/lvname
                2,4,8,16
in the above case if the PE Size is 4MB we can done the extension by

lvextend -l +10 /dev/vgname/lvname

In this setup
# lvextend -L +300M /dev/vg-ctechz/lvm-ctechz
  Extending logical volume lvm-ctechz to 900.00 MB
  Logical volume lvm-ctechz successfully resized


4.Then format this extended size to ext3
     # resie2fs /dev/vgname/lvname

# resize2fs /dev/vg-ctechz/lvm-ctechz
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/vg-ctechz/lvm-ctechz is mounted on /lvm-ctechz; on-line resizing required
Performing an on-line resize of /dev/vg-ctechz/lvm-ctechz to 230400 (4k) blocks.
The filesystem on /dev/vg-ctechz/lvm-ctechz is now 230400 blocks long.

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              25G  4.9G   19G  21% /
tmpfs                 593M     0  593M   0% /dev/shm
/dev/mapper/vg--ctechz-lvm--ctechz
                      885M  542M  301M  65% /lvm-ctechz


Earlier only 20MB is available and 97% full now we extended the lvm and 301MB available and 65% Free now.



# vgs
  VG        #PV #LV #SN Attr   VSize VFree
  vg-ctechz   2   1   0 wz--n- 1.04G 164.00M

# lvs
  LV         VG        Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  lvm-ctechz vg-ctechz -wi-ao 900.00M
 

Now the total lvm size is 900MB

# lvdisplay
  --- Logical volume ---
  LV Name                /dev/vg-ctechz/lvm-ctechz
  VG Name                vg-ctechz
  LV UUID                jZEuoN-16MG-30eX-SZaI-8ETO-ZguH-YRVyeN
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                900.00 MB
  Current LE             225
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0