Wednesday, January 11, 2012

Strace To Debug And Trace Linux System Calls

A few days ago one the applications on my server was constantly crashing but because of God monitoring framework, it was coming back to life. Only thing which was changing was pid of the process. So we ran strace to check out what all system calls are being executed by process. Assuming that process crashing was python script
ps aux | grep python
strace -p <pid_from_above>


This will show you all the system calls being executed. In my case it was SIGKILL which was killing the process. Actually god itself was executing it since the process was detaching from it and was trying to run as a daemon.

You can use strace for the following use cases:
  • To check the system calls done by a command. This is helpful to know what all libraries the binary is trying to access.
    strace <command>
    Run strace echo hello for fun and check out the output.
  • You can capture the output of strace to a file by passing -o flag and then use grep for analysis.
    strace -o output.txt ping 8.8.8.8
  • Another use case for strace is when any of your application is running unexpectedly slow. Just pass -c flag to strace and you'll get statistics of all the system calls executed. You can also pass -p with -c to supply a pid.
    strace -c ping 8.8.8.8
I recommend you to read Solutions for tracing UNIX applications at IBM Developer Works for a more detailed tutorial.

Wednesday, December 14, 2011

IP Packet Filtering: iptables Explained For Beginners

iptables is a IP Filter which is shipped with kernel. Technically speaking, an IP filter will work on Network layer in TCP/IP stack but actually iptables work on data link and transport layer as well. In a broad sense, iptables consists on tables, which consists of chain which is further comprised of rules.
Default tables are:
  1. Raw
  2. Mangle
  3. NAT
  4. Filter
Default chains are (yes, they are written in upper case):
  1. PREROUTING: used by raw, mangle and nat tables
  2. INPUT: used by mangle and filter tables
  3. FORWARD: used by mangle and filter tables
  4. OUTPUT: used by raw, mangle, nat and filter tables
  5. POSTROUTING: used by mangle and nat tables
I'll discuss about Filter table here. It is the one which is most generally used but if you are interested in others as well you can find a detailed tutorial at frozentux. Filter table uses three chains, INPUT, FORWARD and OUTPUT. 
  • INPUT chain is for the packets meant for your own local machine. Reply of a http request made by your browser will go through INPUT chain.
  • OUTPUT chain is for the packets going out of your machine. The http request made by your browser will go through this chain.
  • FORWARD chain is for the packets which you receive but they are not meant for you. Your machine is just supposed to forward them to another device. This generally happens when the machine is configured as a gateway or something similar.
Now every iptables rules have some "target" which is executed when it is matched against a "criteria". Following are the most common targets:
  • ACCEPT: Packet is accepted and goes to the application for processing.
  • DROP: Packet is dropped. No information regarding the drop is sent to the sender.
  • REJECT: Packet is dropped and information (error) message is sent to the sender.
  • LOG: Packet details are sent to syslogd for logging. 
  • DNAT: Rewrites the destination IP of the packet
  • SNAT: Rewrites the source IP of the packet
First four are used in Filter tables a lot. Now let us discuss some of the common criteria:
  • -p <protocol>: It matches protocols like tcp, udp, icmp and all
  • -s <ip_addr>: It matches source IP address
  • -d <ip_addr>: It matches destination IP address
  • --sport <port>: It matches the source port
  • --dport <port>: It matches the destination port
  • -i <interface>: It matches the interface from which the packet entered
  • -o <interface>: It matches the interface from which the packet exits
Now we know the basic things to start building our rules. Let us try to write some rules for a few hypothetical (or real) situations. First we'll set default policy for iptables filter table using -P flag.
iptables -t filter -P INPUT DROP
iptables -t filter -P OUTPUT DROP


Now we'll allow this machine to send only http requests and ssh requests:
iptables -A INPUT -p tcp -i eth0 --dport 80
iptables -A INPUT -p tcp -i eth0 --dport 22

Note that -A flag is used because we want to append these rules to current iptables config. If we do not use -A then the rules will be overwritten.

Also assume that my machine acts as an ftp server so I should allow connections to port 20 and 21.
iptables -A OUTPUT -p tcp -i eth0 --sport 20
iptables -A OUTPUT -p tcp -i eth0 --sport 21


Also I would say that I trust my internal network:
iptables -A OUTPUT -j ACCEPT -p all -d 192.168.1.0/24 -o eth0
iptables -A INPUT -j ACCEPT -p all -s 192.168.1.0/24 -i eth0

So this is how my final iptables script would look like:

iptables -t filter -P INPUT DROP
iptables -t filter -P OUTPUT DROP
iptables -A INPUT -p tcp -i eth0 --dport 80
iptables -A INPUT -p tcp -i eth0 --dport 22
iptables -A OUTPUT -p tcp -i eth0 --sport 20
iptables -A OUTPUT -p tcp -i eth0 --sport 21
iptables -A OUTPUT -j ACCEPT -p all -d 192.168.1.0/24 -o eth0
iptables -A INPUT -j ACCEPT -p all -s 192.168.1.0/24 -i eth0

Note that rules are read sequentially. So other than policy rules, if any rule matches then reading will stop and action will be taken according to that rule.

Sunday, December 4, 2011

Controlling Your Linux Server Using Twitter

Last weekend I wrote about "Using Twitter To Monitor Your Linux Server" using a command line client "MYST". I have added a new feature to MYST using which you can tell your server to execute commands by using DMs. Let us start configuration for the same.

Step 1: Ensure that you can send direct messages to your server's twitter account. Just follow the server's account and your server's account also needs to follow you.

Step 2: Add the following in your .myst.conf file below MYST section:

[Users]
auth_users: adityapatawari 

You can add more twitter handles separated by blank space. 

Step 3: Now you can DM any command to your server's account and it'll be executed. You need to ensure that myst.py script's "getdm" option is in cron.

MYST uses python's os.system to execute these commands. The user's access level who executed myst.py will decide if the commands coming from DM will be executed or not. If you want everything to be executed then run myst.py with super user or using sudo.

Please contribute to the MYST project on Gitorious.org

Sunday, November 27, 2011

Using Twitter To Monitor Your Linux Server!

Yes, you can use Twitter for monitoring your server. I won't say that it is a complete monitoring solution nor I will ask you to throw away your existing monitoring mechanisms. In fact the script I am talking about, MYST (AGPLv3), was not created for this purpose. It was created so that I could tweet without using browser with inspiration from Hiemanshu, a Fedora contributor, using python-twitter api.

It is just a fun script which you can use to tweet the health of your server periodically to a private account which only a moderated set of people can follow. So here is how you do it:

Step 1: Create a Twitter account. From settings page, mark it private.

Step 2: Open Twitter's new application page and fill the form. Put the name as 'MYST' and website as 'http://myst.adityapatawari.com'.

Step 3: Download 'MYST: Twitter for Shell' and extract it.

Step 4: Open your application 'MYST' listed at Twitter's apps page and fill .myst.conf with the relevant details.
Copy it to your home directory.

Step 5: Install python-twitter (version 0.82) on your server along with dependencies.

Step 6: Put a cron with appropriate time (and path to scripts) to execute the following periodically:

./myst.py update `./monitor.sh`

This is a cool method to check out the system health and you can modify monitor.sh to add more parameters to monitor.
Please contribute to the MYST project on Gitorious.org

Wednesday, November 2, 2011

How To Install And Configure Puppet (Getting Puppet Up)

Puppet is a system management tool used my many large and small enterprises to manage their infrastructure. From the top of my head Twitter, Wikipedia, Digg, Nokia, Rackspace are some of the companies using it and there is no reason that you cannot use it to manage that single server you love or the entire data center you own.

Installing puppet is not difficult but i"ll recommend installing puppet on Fedora/EL from tmz repo instead of the official repo. Official repo is generally out of date while tmz repo has latest builds. I don't know if some such repo exists for debian-like distributions. If you are interested in installing from source then you should check out this page. Puppet follows client - server architecture. So you need to run a server and a client (which can be on the same machine). Install "puppet" on the client side and "puppet-server" on the server side.

Now let us start with building the server:

Puppet Server
Step 1: Configure tmz repo and install puppet-server

yum install puppet-server

Step 2: Puppet, by default, looks for site.pp file in manifest directory. Let us create one, if not present already.
mkdir /etc/puppet/manifests
touch /etc/puppet/manifests/site.pp

Step 3: Start the puppet master (a.k.a. server) using:
service puppetmaster start

Puppet Client
Step 1: Tell the client where is the server by adding server entry in [main] section:
[main]
server=puppet.aditya.pa

Step 2: Start the puppet client
service puppet start

Puppet client will request a certificate from master. Now let us go to the master and sign the certificate.
You can check out all the signing requests by firing following on puppet master:
puppet cert --list

Sign the correct certificate by:
puppet cert --sign fed1.aditya.pa

Our puppet is up and running and ready to use. I'll build some manifests and modules to manage applications in next post or if you want, you can catch me at Fedora Users and Developers Conference at Pune, India on 6 Novemeber, 2011 where I'll build some manifests and modules live as a part of hackfest event.