Logrotate: The Most Basic Log Management Tool [Examples]

Logrotate is the default and easiest log management tool around. It is shipped by default with most of the major Linux distributions. Logrotate can help you to rotate logs (in other words, it can create a separate log file per day/week/month/year or on the basis of size of log file). It can compress the older log files. It can run custom scripts after rotation. It can rename the log to reflect the date.

Logrotate scripts goes to /etc/logrotate.d/. Let us see some examples to understand it better. Here we'll rotate /var/log/anyapp.log

1. Rotate logs daily:
$ cat /etc/logrotate.d/anyapp
/var/log/anyapp.log {
daily
rotate 7
} 

The logrotate script above will rotate /var/log/anyapp.log everyday and it'll keep the last 7 rotated log files. Instead of daily you can use monthly or weekly also.

2. Compress the rotated logs:
$ cat /etc/logrotate.d/anyapp
/var/log/anyapp.log {
daily
rotate 7
compress
} 

Now you'll find that logrotate is also compressing the rotated files. This is really a big life saver if you want to save some disk space which is a very common use case specially in VPS or cloud environment.
By default logrotate does a gzip compression. You can alter this behavior by using compresscmd. For example "compresscmd /bin/bzip2" will get you bzip2 compression.

3. Compress in the next cycle:
$ cat /etc/logrotate.d/anyapp
/var/log/anyapp.log {
daily
rotate 7
compress
delaycompress
} 

This is useful in case it is not possible to immediately compress the file. This happens when the process keeps on writing to the old file even after the rotation. If the last line sounded strange to you then you might want to read about inodes. Also note that "delaycompress" will work only if "compress" is included in the script.

4. Compressing the copy of the log:
$ cat /etc/logrotate.d/anyapp
/var/log/anyapp.log {
daily
rotate 7
compress
delaycompress
copytruncate
} 

Copytruncate comes handy in the situation where process writes to the inode of the log and rotating the log might cause process to go defunct or stop logging or a bunch of other issues. Copytruncate copies the log and the further processing is done on the copy. It also truncates the original file to zero bytes. Therefore the inode of the file is unchanged and process keeps on writing to the log file as if nothing has happened.

5. Don't rotate empty log and don't give error if there is no log:
$ cat /etc/logrotate.d/anyapp
/var/log/anyapp.log {
daily
rotate 7
compress
delaycompress
copytruncate
notifempty
missingok
} 

Self explanatory. Both "notifempty" and "missingok" has opposite twins named "ifempty" and "nomissingok" which are the defaults for logrotate.

6. Execute custom script before and/or after logrotation:
$ cat /etc/logrotate.d/anyapp
/var/log/anyapp.log {
daily
rotate 7
prerotate
    /bin/myprescript.sh
endscript
postscript
    /bin/mypostscript.sh
endscript
}

You can run multiple scripts/commands as long as they are in between (pre|post)rotate and endscript. I have removed some of the parameters from the script to maintain readability.

I have just scratched the surface of logrotate. In practice it is capable of much more. You should check out logrotate's man page for more options.

Tcpdump: Packet Filtering And Analysis with Examples

Below are some examples which will help you to understand packet filtering better. Again, a word of warning, before performing packet filtering, make sure that you do not run it in promiscuous mode and you have permissions from your employer or you run it in your own network, better if you could create your own network using virtual machines. Make sure that it is legal in your country.

Let us start.

1. Capture everything coming to your machine and write it to dump.pcap:
tcpdump -w dump.pcap

2. Capture only tcp packets and write it to dump.pcap:
tcpdump tcp -w dump.pcap
You can also use udp in place of tcp to capture udp packets

3. Capture all packets from port 8080 and write it to dump.pcap:
tcpdump port 8080 -w dump.pcap
You can use dst port and src port in place of port to specify the filter for destination and source port only.

4. Capture all udp packets with destination port 53 and write it to dump.pcap:
tcpdump udp and dst port 53 -w dump.pcap
Use and to connect two or more types of filters

5. Capture all the packets from interface eth0 only and write it to dump.pcap:
tcpdump -i eth0

6. Capture all the packets coming from 192.168.1.1:
tcpdump src 192.168.1.1 -w dump.pcap

7. Capture all the packets coming from or going to 192.168.1.1:
tcpdump host 192.168.1.1 -w dump.pcap

8. Capture tcp packets from interface wlan0 going to 192.168.1.2 at port 80:
tcpdump -i wlan0 and tcp and dst 192.168.1.2 and port 80 -w dump.pcap

9. Change the size of packet capture:
tcpdump -s 0 -w dump.pcap
Here -s signifies the snap length. You can specify the maximum length of packets to capture. Bigger packets will be captured in truncated manner. Do not specify the highest limit when you don't have to. Taking larger snapshots both increases the amount of time it takes to process packets and, effectively, decreases the amount of packet buffering. Maximum limit as of now is 65535 with 0 representing the same.

10. Capture only 500 packets:
tcpdump -c 500 -w dump.pcap

11. Capture all the packets except coming from or going to 192.168.1.1:
tcpdump not host 192.168.1.1

12. Capture all the packets coming from and going to entire subnet:
tcpdump net 192.168.1.0/24

Next, you can use wireshark to analyse the packets. I wrote a post sometime ago, about wireshark which might help you.

Scaling Puppet With Apache And mod_passenger

As you keep on adding more and more machines to Puppet, it tends to get slower. A major reason for this is that Puppet uses Webrick by default which it ships with. While webrick is good for testing and small scale deployments, it performs poorly as the number of machines increases. So a good alternative is to use Apache or Nginx in combination with mod_passenger or Unicorn. I'll show you how to use Apache + mod_passenger in a very concise way. I think using other combinations should be equally easy.

Install httpd and mod_passenger into your puppetmaster box. Use Phusion Passenger RPM Repository for mod_passenger if it is not available for your distribution. 

# yum install mod_passenger httpd

Now you got to configure passenger. Include the passenger module in apache config and set various params. To keep the configs clean, I recommend creating a separate file and putting passenger related things there. My passenger config looks like this.

Finally let us create a virtual host for the puppet master. Remember that not only apache has to serve catalogs but it also has to take care of SSL as well. So you need to point apache to the certs and the CA created by the puppet master. You might need to install mod_ssl for turning on the SSLEngine. You can check out my apache virtual host config here. RequestHeader(s) are used to set the certificate verification result as environment variable. 

You can tweak around a bit, specially with passenger config to suit your infra needs. If at all anything is not clear please ask me in comments.

Puppet And Common Errors

Installing Puppet can be a nightmare at times especially if you are doing it for the first time. Error messages are not always obvious and would require some experience to understand. So this is my attempt to explain the errors and suggest the solutions.

Needless to say that step one would always be to ensure that the names are resolving and the puppet client and master can communicate. Also make sure that port 8140 is white listed. 

Error 1: err: Could not request certificate: getaddrinfo: Name or service not known

Probable Solution: Puppet client is not able to reach the puppet master. This usually happens when you are setting up a new environment and puppet master's name is not resolvable. If you can, put a relevant entry in your DNS and add a server variable in [agent] section in puppet.conf. Alternatively you can use /etc/hosts to point the client to the master but you'll have to add appropriate entries on the /etc/hosts of both the puppet master and client.

Error 2: Starting puppetmaster: Could not prepare for execution: Could not find a default provider for user

Probable Solution: This happens because of SELinux restrictions. You can fix this by running a "setenforce 0" which will turn off the SELinux. This is required for CA creation only. So you can turn on SELinux after the puppet master creates CA successfully.

Error 3: err: Could not request certificate: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key

Probable Solution: Looks like your certificates have gone bad. You should remove /var/lib/puppet/ssl directory and request for new certs signed by puppet master.

Error 4: err: Could not retrieve catalog from remote server: hostname was not match with the server certificate

Probable Solution: This may happen if you are referring to the puppet master by a wrong name. In other words, the CA is not built to use this name. You can check out the correct CA name in the file /var/lib/puppet/ssl/ca/inventory.txt. You should put this name in the [agent] section assigned to server variable.

Error 5: err: Could not retrieve catalog from remote server: Connection refused - connect(2)

Probable Solution: This is happening because your puppet client is not able to connect to puppet master. One reason might be firewall which is rejecting the packets and the other reason might be that puppet master has died. So you either need to relax your firewall or make sure that your puppet master is always up and running. You may want to use daemontools or god or a similar application.

Error 6: Exiting; no certificate found and waitforcert is disabled

Probable Solution: This usually happens when a new node is introduced in the infrastructure. Issue is that this node do not have the certificate yet and since "--waitforcert" flag was not enabled, it exited immediately. If your puppet master has autosign enabled that just add the flag "--waitforcert X" with X replaced with time in seconds like 60. If autosign is not enabled then you have to sign the cert for the client manually at your puppet master.

I'll add more as I encounter them. Please let me know in comments if I am wrong anywhere. Have fun with Puppet :)

Strace To Debug And Trace Linux System Calls

A few days ago one the applications on my server was constantly crashing but because of God monitoring framework, it was coming back to life. Only thing which was changing was pid of the process. So we ran strace to check out what all system calls are being executed by process. Assuming that process crashing was python script
ps aux | grep python
strace -p <pid_from_above>

This will show you all the system calls being executed. In my case it was SIGKILL which was killing the process. Actually god itself was executing it since the process was detaching from it and was trying to run as a daemon.

You can use strace for the following use cases:

• To check the system calls done by a command. This is helpful to know what all libraries the binary is trying to access.
  strace <command>
  Run strace echo hello for fun and check out the output.
• You can capture the output of strace to a file by passing -o flag and then use grep for analysis.
  strace -o output.txt ping 8.8.8.8
• Another use case for strace is when any of your application is running unexpectedly slow. Just pass -c flag to strace and you'll get statistics of all the system calls executed. You can also pass -p with -c to supply a pid.
  strace -c ping 8.8.8.8

I recommend you to read Solutions for tracing UNIX applications at IBM Developer Works for a more detailed tutorial.