How to setup Logging Remotely (on a remote machine)

Problem: You want system logger messages saved on a remote machine rather than locally
 
Solution: Configure /etc/syslog.conf for remote logging, using the "@" syntax:

/etc/syslog.conf
# send all messages to remote system "loghost"
 
*.*    @loghost

On loghost, tell syslogd to accept messages from the network by adding the -r option:
 
# syslog -r ....

or within /etc/sysconfig/syslog:

SYSLOGD_OPTIONS=".... -r ...."

Remember to send a signal to syslogd to pick up any changes to /etc/syslog.conf or restart he demon on loghost(remote host[above exmp]). If we store it in remote machine if an intruder breals in the sytem also he cannt remove the log messages or tampered it.

The system logger will not accept messages from another machine by default. To allow this add the syslogd -r command-line option on loghost.

To allow the loghost to be changed easily, set up a "loghost" CNAME record on your nameserver that points to a specific machine:

loghost IN CNAME watchdog.example.com.

How to setup Apache Cassandra

* Cassandra is a highly scalable, eventually consistent, distributed, structured key-value store.

* Cassandra brings together the distributed systems technologies from Dynamo and the data model from Google's BigTable.

* Like BigTable, Cassandra provides a ColumnFamily-based data model richer than typical key/value systems.

1. Install Java
    Java >= 1.6 (OpenJDK and Sun have been tested)

This short guide will walk you through getting a basic one node cluster up and running, and demonstrate some simple reads and writes.

  * tar -zxvf apache-cassandra-$VERSION.tar.gz
  * cd apache-cassandra-$VERSION
  * sudo mkdir -p /var/log/cassandra
  * sudo chown -R `whoami` /var/log/cassandra
  * sudo mkdir -p /var/lib/cassandra
  * sudo chown -R `whoami` /var/lib/cassandra

The sample configuration files in conf/ determine the file-system locations Cassandra uses for logging and data storage. You are free to change these to suit your own environment and adjust the
 path names used here accordingly.

Now that we're ready, let's start it up!

  *  bin/cassandra -f
  *  /usr/local/cassandra/bin/cassandra -f & -------- replace with your locations, where you install cassandra.

# ./cassandra ------ this is just enough to start cassandra, after that to start cassandra-command-line interface to read or write data just run below command

Unix: Running the startup script with the -f argument will cause Cassandra to remain in the foreground and log to standard out.

Now let's try to read and write some data using the command line client. Cassandra ships with a very basic interactive command line interface. Using the CLI you can connect to remote nodes in the cluster to create or update your schema and set and retrieve records.

*  bin/cassandra-cli --host localhost
# ./cassandra-cli --host 192.168.1.67  ------------------- If it is connected correctly it will display the below messages,
Connected to: "Test Cluster" on 192.168.1.67/9160
Welcome to Cassandra CLI version 1.0.7

Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.

[default@unknown]
The command line client is interactive so if everything worked you should, be sitting in front of a prompt...

  Connected to: "Test Cluster" on localhost/9160
  Welcome to cassandra CLI.

As the banner says, you can use 'help;' or '?' to see what the CLI has to offer, and 'quit;' or 'exit;' when you've had enough fun.


  [default@unknown] create keyspace Keyspace1;
  ece86bde-dc55-11df-8240-e700f669bcfc
  [default@unknown] use Keyspace1;
  Authenticated to keyspace: Keyspace1
  [default@Keyspace1] create column family Users with comparator=UTF8Type and default_validation_class=UTF8Type and key_validation_class=UTF8Type;
  737c7a71-dc56-11df-8240-e700f669bcfc

  [default@KS1] set Users[jsmith][first] = 'John';
  Value inserted.
  [default@KS1] set Users[jsmith][last] = 'Smith';
  Value inserted.
  [default@KS1] set Users[jsmith][age] = long(42);
  Value inserted.
  [default@KS1] get Users[jsmith];
  => (column=last, value=Smith, timestamp=1287604215498000)
  => (column=first, value=John, timestamp=1287604214111000)
  => (column=age, value=42, timestamp=1287604216661000)
  Returned 3 results.

If your session looks similar to what's above, congrats, your single node.
 
cluster is operational! But what exactly was all of that? Let's break it down into pieces and see.

set Users[jsmith][first] = 'John';
        \      \        \          \
         \      \_ key   \          \_ value
          \               \_ column
           \_ column family

Data stored in Cassandra is associated with a column family (Users), which in turn is associated with a keyspace (Keyspace1). In the example above, we set the value 'John' in the 'first' column for
key 'jsmith'.

Configuration

cassandra.yaml: main Cassandra configuration file
log4j-server.proprties: log4j configuration file for Cassandra server.

Optional configuration files

cassandra-topology.properties: used by PropertyFileSnitch


cd /opt/apache-cassandra-1.0.7/conf/
 
vim cassandra.yaml
- seeds: "192.168.1.67"

# - seeds: "cassandra1.ctechz.com,cassandra2.ctechz.com,cassandra3.ctechz.com"

# Setting this to 0.0.0.0 is always wrong.
listen_address: 192.168.1.67
# listen_address: cassandra1.ctechz.com


# Leaving this blank has the same effect it does for ListenAddress,
# (i.e. it will be based on the configured hostname of the node).
rpc_address: 192.168.1.67
# rpc_address: cassandra1.ctechz.com

How to setup log files in linux

Directing System Messages to Log Files (syslog)

Problem: You want to configure the system logger to use an organized collection of log files,
 
Solution: Setup /etc/syslog.conf for local logging

# vim /etc/syslog.conf

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                      /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none  /var/log/messages

# The authpriv file has restricted access.
authpriv.*                             /var/log/secure

# Log all the mail messages in one place.
mail.*                                 -/var/log/maillog

# Log cron stuff
cron.*                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                      *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                       /var/log/spooler

# Save boot messages also to boot.log
local7.*                               /var/log/boot.log

After you modify /etc/syslog.conf you must send a signal to force syskogd to reread it and apply your changes, Any of these will do

# kill -HUP `pidof syslogd`
OR
# /etc/init.d/syslog reload
OR
# service syslog reload

How to setup snort in linux

Problem: you want to set up snort, a network-intrusion detection system(NIDS)

Installation:
# tar -xvjf snort-*.tar.gz
# cd snort
# ./configure
# make
# make install

next create a logging directory, if it should not be publically readable, since it will contain potentially sensitive data,

# mkdir -p -m go-rwx /var/log/snort

Finally, install the configuration files and rules database:

# mkdir -p /usr/local/share/rules
# cp etc/* rules/* .rules /usr/local/share/rules


Packet sniffing with snort

To format and print network trace information
# snort -v [-d|-X] [-C] [-e] [filter-expression]

To sniff packets from the network:
# snort [-i interface] [-P snap-length] [filter-expression]

To read network trace data you hae saved previously
# snort -r filename [filter-expression]

Detecting intrusions with snort

Problem: You want to notice if your system is under attack from the network

Solution: To run as a network intrusion detection system, with binary logging, and alerts sent to the system logger:

# snort -c /usr/local/share/rules/snort.conf -b -s
To run snort in the background, as a daemon:
# snort -D [-u user] [-g group] [-m umask] -c

snort -c /usr/local/share/rules/snort.conf

The configuration file includes a large number of pattern matching rules that control logging and alerts,

Decoding snort alert messages

Consult the snort signature database at http://www.snort.org/snort-db, using the signature ID as an index, or searching based on the text message. NIDS database @ http://www.whitehats.com

Loggin with snort

want to manage snorts output and log files in a efficient, effective manner

To log network trace data for latter analysis:
# snort -b [-l logging-directory] [-L basename]

To examine the network trace data:
# snort -r logfile

To manage the logs, don't use logrotate. Instead periodically tell snort to close all of its files and restart, by sending it a SIGHUP signal,
# kill -HUP `pidof snort`

Remove all files that are older that a week,
# find /var/log/snort -type f -mtime +7 -printo | xargs -o -r rm

Remove empty sub-directories
# find /var/log/snort -mindepth 1 -depth -type d -printo | xargs -o -r rmdir -v --ignore-fail-on-non-empty

put those in chrone script....

How to Search for strings in Network Traffic

Problem: You want to watch network traffic, searching for strings in the transmitted data

Solution: Use ngrep

To search for packets containging data that matches a regular expression and protocols that match a filter expression:

# ngrep [grep-options] regular-expression [filter-expression]

To search instead for a sequence of binary data:

# ngrep -X hexadecimal-digits [filter-expression]

To sniff packets and save them in a file:

# ngrep -O filename [ -n count] [ -d interface] [-s snap-length] regular-expression [filter-expression]

To read and display the saved network trace data

# ngrep -I filename regular-expression [filter-expression]

Installation: # tar -xvjf ngrep-*tar.gz
# cd ngrep
# ./configure --prefix=/usr/local
# make
 and install it into /usr/local as root
# mkdir -p /usr/local/bin /usr/local/man/man8
# make install

How to Trace a Process

Problem: You want to know what an unfamiliar process is doing

Solution: To attach to a running process and trace system calls,

# strace -p pid

To trace network system calls:

# strace -e trace=network,read,write
# strace -e trace=network,read,write finger katie@server1.example.com

Run strace against /bin/sshd and capture its output to a text file in output.txt:

$ strace -o output.txt /bin/sshd


You can strace the webserver process and see what it's doing. For example, strace apache process, enter:

$ strace -p 22254 -s 80 -o /tmp/debug.httpd.txt

To see only a trace of the open, read system calls, enter:

$ strace -e trace=open,read -p 22254 -s 80 -o debug.apache.txt

-o filename : Write the trace output to the file filename rather than to screen (stderr).

-p PID : Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (hit CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to up to 32 processes in addition to command (which is optional if at least one -p option is given).

-s SIZE : Specify the maximum string size to print (the default is 32).

SVN Most Useful Commands

Subversion is a free/open-source version control system.

A tree of files is placed into a central repository.

The repository is much like an ordinary file server, except that it remembers every change ever made to your files and directories.
This allows you to recover older versions of your code, or examine the history of how your code was changed.

SVN client program which manages local reflections of portions of that versioned data which is called as working copy.

SVN Checkout – Create working copy

Checkout command is used to download sources from SVN repository to working copy.
 
SVN checkout creates the working copy, from where you can do edit, delete, or add contents. You can checkout a file, directory, trunk or whole project.

$ svn checkout/co URL PATH

When you do a checkout, it creates hidden directory named .svn, which will have the repository details.

SVN Commit – Save changes to the repository

Whenever you do changes to the working copy, it will not reflect in SVN server. To make the changes permanent, you need to do SVN commit.

$ svn commit -m "log messages"

SVN List – Lists directory entries

svn list is useful when you want to view the content of the SVN repository, without downloading a working copy.
 
lists all the files available in the given URL in the repository.
  
$ svn list url

When you execute svn list command with –verbose option it displays the following information.

Revision number of the last commit
Author of the last commit
Size (in bytes)
Date and time of the last commit 

$ svn list --verbose https://www.ctechz.com/svn/project/trunk

 16 ctechz  28361  Apr 16 21:11 README.txt
 21 ctechz   0        Apr 18 12:22 INSTALL
 22 ctechz         Apr 18 10:17 src/

SVN Add – Add a new file to SVN repository

When you want to add a new file (or directory) to the repository you need to use SVN add command. 

The repository will have newly added file, only when you do SVN commit. Now let us add a new file called “ctechz” to our repository.

* Create a file in local working copy
touch ctechz

* Add the file into SVN repository

svn add filename will add the files into SVN repository. From working copy

$ svn add ctechz
A         ctechz

* Commit the added file
Until you commit, the added file will not be available in the repository.

$ svn commit -m "Adding file ctechz" projectURL  
  ------- if from working dir it will upload automatically
Adding         ctechz
Transmitting file data .
Committed revision 84.

SVN Delete – Removing a file from repository

SVN delete command deletes an item from the working copy (or repository). File will be deleted from the repository when you do a SVN commit.

$ svn delete URL

$ svn delete ctechz  -- it will remove the recently created file

$ svn commit -m "Deleting ctechz file" ctechz
Deleting       ctechz
Committed revision 85.

SVN Diff – Display the difference

SVN diff displays the differences between your working copy and the copy in the SVN repository. You can find the difference between two revisions and two paths etc.

$ svn diff filename
$ svn -r R1:R2 diff filename

The above example compares the filename@R1 and filename@R2.

Now the content of the file ctechz will be,

# cat ctechz
testing

now edited the content of ctechz file from testing to tester, which is shown below using the svn diff command.

$ svn diff ctechz
Index: ctechz
--------------------------------
--- ctechz   (revision 85)
+++ ctechz   (working copy)
@@ -1 +1 @@
-testing
+tester 

SVN Status – Status of the working copy

Use svn status command to get the status of the file in the working copy. 

It displays whether the working copy is modified, or its been added/deleted, or file is not under revision control, etc.

$ svn status PATH

The following example shows the status of local working copy,

$ svn status /home/ctechz/cfg
M        /home/ctechz/cfg/ftp_user.cfg
M          /home/ctechz/cfg/ctechz 

‘M’ represents that the item has been modified.  Check for “svn help status” for more options.

SVN Log – Display log message

SVN remembers every change made to your files and directories. To know all the commits made in a file or directory, use SVN log command.

$ svn log PATH

$ svn log ctechz(or project path)

SVN Move – Rename file or directory

This command moves a file from one directory to another or renames a file.

$ svn move src dest

The file will be moved on your local working copy immediately (as well as on the repository after committing).

$ svn move ctechz new-ctechz

Now the file is renamed only in the working copy, not in the repository. To make the changes permanent, you need to commit the changes. 

$ svn commit -m "Renaming ctechz to new-ctechz" new-ctechz

SVN Update – Update the working copy

svn update command brings changes from the repository into your working copy. If no revision is specified, it brings your working copy up-to-date with the HEAD revision. Otherwise, it synchronizes the working copy to the revision given in the argument.

Always before you start working in your working copy, update your working copy. So that all the changes available in repository will be available in your working copy. i.e latest changes.

$ svn update PATH

In case some other user added/deleted file in Repository, your working copy will not have those files by default, until you update your working copy.

$ svn update
A  new/user2
A  new/web1
Updated to revision 816