Sunday, August 28, 2011

How To Install and Configure Splunk For Log Analysis

Splunk, in simple words, is a log analyzer, a powerful one. It understands machine data like no one else and make it searchable and displays it in easy to understand way with a web dashboard. This data is important not only for searches but also for investigations, monitoring and making decisions regarding your infrastructure. Large organizations like Motorola and Vodafone use splunk to keep an eye on their server and now I am going to show you, how you can do the same.

How to install Splunk?
To install splunk just download the relevant installer from this page and use a suitable package manager. I'll use Centos 5.6 for this post. So just do a "yum install splunk-4.2.3-105575.i386.rpm". The rpm will be installed in /opt/splunk. Chances are that it won't be in your path so either add it or just use the full path. If the splunk is not in the /opt run a find on it "find / -name splunk". To start splunk server just issue the command " /opt/splunk/bin/splunk start" and accept the license. Don't forget to allow the port, usually 8000, from you firewall.

Adding logs for analysis
Let us add apache logs to the splunk for analysis. These logs are located at /var/log/httpd/. Log in using the web console at http://localhost:8000 with credentials id: admin and password: changeme. Click on the "Add data" button and then on Apache logs link. Now you got to "Consume Apache logs on this Splunk server", click next and specify the path to the apache logs directory and you are done.

Searching through the logs
From home page go to the welcome tab and launch the search app. Now you just need to type the search query and the relevant parts from the logs will be presented to you. 

Building Reports
From the search app only, click on the "Build Report" link and just specify the criteria and that is it. You'll get the report in no time.

And there it is. Splunk is good to go. Explore more and comment here. :)

Monday, August 22, 2011

Munin: Server Statistics, The Easy Way

Server usage statistics always play a useful role when things go wrong. Often they provide you with the warnings like "oops! there is too much data, we'll need a new disk" or "wow! good traffic, maybe we should get more bandwidth". If you are asked to write code yourself for collecting these data and represent in an graphical fashion then you'll spend a few days, if not weeks to do so.

Well, why to reinvent wheel? Just use munin and be happy. Installing munin is easy enough. Just do a yum install munin munin-node and you'll be done.
After installation you need to configure it. For that edit file /etc/munin/munin.conf and uncomment and edit the following lines

dbdir   /var/lib/munin
htmldir /var/www/html/munin
logdir  /var/log/munin
rundir  /var/run/munin
# Where to look for the HTML templates
tmpldir /etc/munin/templates
# a simple host tree
    use_node_name yes

Now start the munin node service service munin-node start. Well, that is it. Just wait for a few minutes and then go to http://localhost/munin to get the graphs.

Here are a couple of the graphs I have obtained (and there are tonnes more):

Friday, August 19, 2011

Monit: Monitor Your Server And Ensure Service Uptime

Many of us (specially sysadmins) often have nightmares of servers going down at odd hours and clients shouting on us. Right from a large organization to people having personal servers just to host a blog, want their servers up and running all the time. Of course, one cannot stay up 24X7 to watch over so we'll do some automation to make sure that the services are running.

Enter Monit! According to their website "Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources. You can use Monit to monitor files, directories and filesystems for changes, such as timestamp changes, checksum changes or size changes. You can also monitor remote hosts; Monit can ping a remote host and can check TCP/IP port connections and server protocols". We'll look into the "start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources" part today.

To install monit just do a yum install monit and you are good to go. Next, you need to configure it. I'll show you the configuration for httpd. Others are similar, you can check out monit wiki for more how-tos.

The config file is located at monit.conf, although you don't have to fiddle with it but give it a read if you want.
We'll be writing and saving our configs in /etc/monit.d/ directory. So, create a file /etc/monit.d/apache and write the following lines in it.

check process httpd with pidfile /var/run/
group apache
start program = "/usr/sbin/httpd -k start"
stop program = "/usr/sbin/httpd -k stop"
if failed host port 80 protocol http
   then restart
if 5 restarts within 5 cycles then timeout

Now I'll explain these lines. First two lines just ask monit to check process named httpd whose pids are written in /var/run/ file and they belong to apache group. Next two lines are telling the start and stop commands for the httpd service. Notice that I have not used "service httpd start". The next two lines tell monit to restart in case it finds that at port 80 http is not running. Lastly, we instruct monit to do this for a maximum of 5 times consecutively and then stop because if in 5 consecutive checks the service keeps on shutting down then problem is something else. I did not use service httpd start/stop because monit requires fully qualified file names to execute.

Next you need to start the monit service by firing service monit start and then you can go to sleep having some peace of mind

My next few posts are going to be on Automation and Network Monitoring only.Enjoy!