Wednesday, October 24, 2012

Common Linux Termination Signals

People issuing a kill -9 on production servers blindly should read this:

If you have ever used Linux terminal, I think you should know what Linux Termination Signals are. They are the signals that you usually supply with kill command when you are trying to kill a process. Depending upon the provided signal further actions can be taken.

  • SIGTERM: It is the default signal used by kill command. This signal can be handled by the process in the sense that process can choose to ignore this or do some specific action upon receiving this signal. It is like politely asking someone to do something.
  • SIGKILL: This is the signal sent when you supply -9 as argument to the kill command. This signal cannot be handled or ignored. Process has to die as a result of this signal. Usually this is the last resort thing. If there is a process which is not dying even after receiving a SIGKILL then you should report it to upstream. It is most likely some sort of bug in distribution or the app itself.
  • SIGHUP: This signal tells the process that user's terminal is disconnected somehow. Usually this happens when you ssh pipe gets broken or you face network connectivity issues etc.
You can get a list of available signals for your distribution by firing kill -l on terminal. For my Fedora 17 box there are 64 signals available. I think the number would be the same for the distributions following POSIX standard.

UPDATE: (As mentioned by @MarcusMoeller on twitter) SIGHUP is commonly used by daemons to reload config files.

Daemontools: To Ensure That Your Processes Are Always Running

In production environment, it is often required to restart the process if it dies. Not doing so fast enough may result in user facing downtime which can impact the business, hence automation of this job is critical to reduce response time.

There are several tools which do similar jobs like Daemontools, Supervisor, bash scripts, Monit etc. We are going to talk about Daemontools since it is very light, implements basic feature set and allows you to do more over its standard commands. It is written by D. J. Bernstein, the creator of djbdns and several awesome tools.

Daemontools runs the processes as its own child process and spawns a new one as soon as the older dies. You have to ensure that the process you are trying to run using daemontools runs in foreground and is not a daemon in itself. The svscan program of Daemontools will scan the present working directory and its subdirectories and launch one supervise process for each run script found.

Installation:
For Fedora, Red Hat and other Enterprise Linux you can use untroubled.org's daemontools rpm. Daemontools itself was written in 11 years ago (and it still works like a charm!) so don't worry about the stale rpms.
For Ubuntu server and similar distributions, you can install deamontools and daemontools-run packages and you would be good to go.

In case you want to do a manual install, instructions can be obtained from the Daemontools website.

Usage:
Once the daemontools is installed, all you need to do is create an executable script named run and put the command to launch the process in the script. Start the svscan tools after this and let the magic happen. 

Let us see this though an example. Say you have 2 services, nginx and php-fpm which you want to restart immediately if they die then here is how your directory structure should look like:
Deamontools install dir (usually /etc/service)
├── nginx
│   └── run
└── php-fpm
    └── run

run file should have a shebang like #!/bin/bash or #!/usr/bin/python defined and it should have executable bit set. Once you are done with this, start svscan either manually by running svscan command or if you have a SysV init script then use that. 

I also suggest you to go through "man svc" to get an idea of commonly used commands of Daemontools.