cut | rev | wc | sort | uniq | diff | tr | ls | find

Cut

Cut out selected fields of each line of a file

Extracting text by column:
# cut -f3 -d: /etc/passwd (display third colon(:)-delimited field)
# cut -c5 /etc/passwd    (display 5th character)
# cut -c1-5 /etc/passwd  (display first 5 characters)

-d  specify the column delimiter (default is TAB)
-f   specify column to print
-c   cut by character

1.Display the 1st field (employee name) from a colon delimited file
# cut -f1 -d" "  names.txt   ---> Here delimiter is space

2.Display 1st and 3rd field from a colon delimited file
# cut -f1,3 -d:  names.txt
# cut -d: -f1,3 /etc/passwd

Emma Thomas:Marketing
Alex Jason:Sales
Madison Randy:Product Development
Sanjay Gupta:Support
Nisha Singh:Sales

3.Display only the first 8 characters of every line in a file
# cut -c1-8 /etc/passwd

oprofile
rpcuser:
nfsnobod
xfs:x:43
haldaemo
avahi-au
gdm:x:42
sabayon:
vboxadd:

4. To print the characters from tenth position to the end, specify only the start position and omit the end position.
# cut -c10- file.txt
 
rev  --- reverse

Reverse lines of a file



First reverse the text in each line and then apply the command on it.
# rev filenames.txt | cut -d'.' -f1

Word Count (wc)
 
The wc (word count) command in Unix/Linux operating systems is used to find out number ofnewline count, word count, byte and characters count in a files specified by the file arguments.

wc -l : Prints the number of lines in a file.
wc -w : prints the number of words in a file.
wc -c : Displays the count of bytes in a file.
wc -m : prints the count of characters from a file.
wc -L : prints only the length of the longest line in a file.

# wc file1              
(displays no. of lines, words and character in file1)

# cat names2.txt
wali
salman
obama
wali
wali
wali
ajay
sameer

# wc  *                 
(displays no. of lines, words and character of every files in the current directory)

-l only for line count
-w only for word count
-c only for byte count
-m only for character count (1 character = 1 byte)
 
 Sort

Sort command is helpful to sort/order lines in text files. You can sort the data in text file and display the output on the screen, or redirect it to a file.

# grep bash /etc/passwd | sort

(sort the UIDs in ascending order)
# sort  -t:  -k3  -n  /etc/passwd    

(shows only UIDs in ascending order)
# sort  -t:  -k3  -n  /etc/passwd | cut  -f3  -d:

-r performs a reverse (descending) sort
-n performs a numeric sort
-f ignores (folds) case of characters in strings
-u (unique) removes duplicate lines in output
-t: uses : as a filed separator
-k3 third column by : delimited field
  
uniq

Uniq command is helpful to remove or detect duplicate entries in a file. 

1. Eliminating duplicate lines:

cat>file
Emma Thomas:Marketing
Alex Jason:Sales
Madison Randy:Product Development
Sanjay Gupta:Support
Nisha Singh:Sales
wali
salman
obama
wali
wali
wali
ajay
sameer

# uniq file
(uniq without argument, removes duplicate adjacent lines)

Emma Thomas:Marketing
Alex Jason:Sales
Madison Randy:Product Development
Sanjay Gupta:Support
Nisha Singh:Sales
wali
salman
obama
wali
ajay
sameer

-u to output only the lines that are truly unique, only occurring once in the input.

# uniq -u names.txt

Emma Thomas:Marketing
Alex Jason:Sales
Madison Randy:Product Development
Sanjay Gupta:Support
Nisha Singh:Sales
wali
salman
obama
ajay
sameer

-d to output only print one copy of the lines that are repeated in the input, Duplicate lines.

# uniq -d names.txt
wali

-c each line will be prepended with a number indicating how many times it appears in the input.
 
# uniq -c names.txt
      1 Emma Thomas:Marketing
      1 Alex Jason:Sales
      1 Madison Randy:Product Development
      1 Sanjay Gupta:Support
      1 Nisha Singh:Sales
      1 wali
      1 salman
      1 obama
      3 wali
      1 ajay
      1 sameer

Comparing files (diff)

Displays two files and prints the lines that are different.

# diff  file  file1

# diff  -u  file  file1  
(line that begin with + exist in names2.txt but not in names.txt,
line that begin with - exist in names.txt but not in names2.txt)

# diff -u names.txt names2.txt
--- names.txt   2013-11-02 00:15:44.000000000 +0530
+++ names2.txt  2013-11-02 00:25:29.000000000 +0530
@@ -1,8 +1,3 @@
-Emma Thomas:Marketing
-Alex Jason:Sales
-Madison Randy:Product Development
-Sanjay Gupta:Support
-Nisha Singh:Sales
 wali
 salman
 obama

# Colordiff file1 file2

 tr
 
tr is an UNIX utility for translating, or deleting, or squeezing repeated characters. It will read from STDIN and write to STDOUT.

1. Convert lower case to upper case

# tr abcdefghijklmnopqrstuvwxyz ABCDEFGHIJKLMNOPQRSTUVWXYZ

# tr a-z A-Z
thegeekstuff
THEGEEKSTUFF

2. Translate braces into parenthesis

You can also translate from and to a file. In this example we will translate braces in a file with parenthesis.

$ tr '{}' '()' < inputfile > outputfile

The above command will read each character from “inputfile”, translate if it is a brace, and write the output in “outputfile”.

3. Translate white-space to tabs

The following command will translate all the white-space to tabs

$ echo "This is for testing" | tr [:space:] '\t'
This    is    for    testing

4. Squeeze repetition of characters using -s

 you can convert multiple continuous spaces with a single space

$ echo "This  is  for testing" | tr -s [:space:] ' '
This is for testing

5. Delete specified characters using -d option

tr can also be used to remove particular characters using -d option.

$ echo "the geek stuff" | tr -d 't'
he geek suff

To remove all the digits from the string, use

$ echo "my username is 432234" | tr -d [:digit:]
my username is

6. Complement the sets using -c option

You can complement the SET1 using -c option. For example, to remove all characters except digits, you can use the following.

$ echo "my username is 432234" | tr -cd [:digit:]
432234

7. Join all the lines in a file into a single line

The below command will translate all newlines into spaces and make the result as a single line.

$ tr -s '\n' ' ' < file.txt

 LS

Listing files and directorie's

1. -t  sort by modification time

2. Display One File Per Line Using # ls -1
    Display All Information About Files/Directories # ls -l

 -rw-r--r-- 1 root root 6066 Oct 28 12:53 install.log.syslog
 drwxr-xr-x 4 root root 4096 Nov 1 12:04 j

1st Character – File Type: First character specifies the type of the file.

In the example above the hyphen (-) in the 1st character indicates that this is a normal file. Following are the possible file type options in the 1st character of the ls -l output.

    Field Explanation
    - normal file
    d directory
    s socket file
    l link file
   
Field 1 – File Permissions
Field 2 – Number of links: Second field specifies the number of links for that file. In this example, 1 indicates only one link to this file.
Field 3 – Owner
Field 4 – Group
Field 5 – Size
Field 6 – Last modified date & time
Field 7 – File name

3. Display File Size in Human Readable Format Using 
    # ls -lh

4. Display Directory Information Using 
   # ls -ld

5. Order Files Based on Last Modified Time Using 
   # ls -lt

6. Order Files Based on Last Modified Time (In Reverse Order) Using
   # ls -ltr

7. Display Hidden Files Using 
   # ls -a (or) # ls -A

It will show all the files including the ‘.’ (current directory) and ‘..’ (parent directory).
 
To show the hidden files, but not the ‘.’ (current directory) and ‘..’ (parent directory), use option -A.
     
8. Display Files Recursively Using # ls -R

# ls -R
.:
anaconda-ks.cfg  Desktop  install.log  install.log.syslog  ipt  j  lvm-sizefile  lvmsnap1.tar

./Desktop:
LVM  LVM~

./j:
32Bit  AUTORUN.INF  VBoxLinuxAdditions-amd64.run  VBoxSolarisAdditions.pkg        VBoxWindowsAdditions.exe
64Bit  autorun.sh   VBoxLinuxAdditions-x86.run    VBoxWindowsAdditions-amd64.exe  VBoxWindowsAdditions-x86.exe

./j/32Bit:
OS2  Readme.txt

./j/32Bit/OS2:
gengradd.dll  libc063.dll  readme.txt  VBoxGuest.sys  vboxmouse.sys  VBoxService.exe

./j/64Bit:
Readme.txt

It will go through all dir in the current location and display the contents.


9. Display File Inode Number/iNode number Using 
  # ls -i

10. Display File UID and GID Using 
   # ls -n

11. Visual Classification of Files With Special Characters Using 
  # ls -F

Find

Find the passwd file under root and two levels down
# find / -maxdepth 3 -name passwd

Min depth and Max depth
# find -mindepth 3 -maxdepth 5 -name passwd

Inverting the match
# find -maxdepth 1 -not -iname "MyCProgram.c"

Find file by inode number
# find -inum 16187430

Find files which has read permission only to group
# find . -perm g=r -type f -exec ls -l {} \;

Find files which has read permission only to group
# find . -perm 040 -type f -exec ls -l {} \;

Find all empty files (zero byte file) in your home directory and its sub-directory# find ~ -empty

List all the empty files only in your home directory.
# find . -maxdepth 1 -empty

List only the non-hidden empty files only in the current directory.
# find . -maxdepth 1 -empty -not -name ".*"

Finding the Top 5 Big Files
# find . -type f -exec ls -s {} \; | sort -n -r | head -5

Finding the Top 5 Small Files. Technique is same as finding the bigger files, but the only difference the sort is ascending order.# find . -type f -exec ls -s {} \; | sort -n  | head -5

---> Find Files Based on file-type using option -type:-

Find only the socket files.
# find . -type s

Find all directories
# find . -type d

Find only the normal files
# find . -type f

Find all the hidden files
# find . -type f -name ".*"

Find all the hidden directories
# find -type d -name ".*"

---> Find Files by Size

Find files bigger than the given size
# find ~ -size +100M

Find files smaller than the given size
# find ~ -size -100M

Find files that matches the exact given size
# find ~ -size 100M

---> Remove big archive files using find command

The following command removes *.zip files that are over 100M.
# find / -type f -name *.zip -size +100M -exec rm -i {} \;"

---> Find files whose content got updated within last 1 hour
 -mmin n File’s data was last modified n minutes ago
 -mtime n File’s data was last modified n*24 hours ago

Find files in the current directory and sub-directories, whose content got updated within last 1 hour (60 minutes) # find . -mmin -60

Finds all the files (under root file system /) that got updated within the last 24 hours (1 day).# find / -mtime -1

---> Find files which got accessed before 1 hour
 -amin n File was last accessed n minutes ago
 -atime n File was last accessed n*24 hours ago

Find files in the current directory and sub-directories, which got accessed within last 1 hour (60 minutes)# find -amin -60

Finds all the files (under root file system /) that got accessed within the last 24 hours (1 day).# find / -atime -1

---> Find files which got changed exactly before 1 hour
 -cmin n File’s status was last changed n minutes ago.
 -ctime n File’s status was last changed n*24 hours ago.
Find files in the current directory and sub-directories, which changed within last 1 hour (60 minutes)# find . -cmin -60

Finds all the files (under root file system /) that got changed within the last 24 hours (1 day).# find / -ctime -1

---> Long list the files which are edited within the last 1 hour.
# find -mmin -60
./cron
./secure

# find -mmin -60 -exec ls -l {} \;
-rw-------  1 root root 1028 Jun 21 15:01 ./cron
-rw-------  1 root root 831752 Jun 21 15:42 ./secure