Enter Monit! According to their website "Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources. You can use Monit to monitor files, directories and filesystems for changes, such as timestamp changes, checksum changes or size changes. You can also monitor remote hosts; Monit can ping a remote host and can check TCP/IP port connections and server protocols". We'll look into the "start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources" part today.
To install monit just do a yum install monit and you are good to go. Next, you need to configure it. I'll show you the configuration for httpd. Others are similar, you can check out monit wiki for more how-tos.
Now I'll explain these lines. First two lines just ask monit to check process named httpd whose pids are written in /var/run/httpd.pid file and they belong to apache group. Next two lines are telling the start and stop commands for the httpd service. Notice that I have not used "service httpd start". The next two lines tell monit to restart in case it finds that at port 80 http is not running. Lastly, we instruct monit to do this for a maximum of 5 times consecutively and then stop because if in 5 consecutive checks the service keeps on shutting down then problem is something else. I did not use service httpd start/stop because monit requires fully qualified file names to execute.
Next you need to start the monit service by firing service monit start and then you can go to sleep having some peace of mind
My next few posts are going to be on Automation and Network Monitoring only.Enjoy!