Strace To Debug And Trace Linux System Calls

A few days ago one the applications on my server was constantly crashing but because of God monitoring framework, it was coming back to life. Only thing which was changing was pid of the process. So we ran strace to check out what all system calls are being executed by process. Assuming that process crashing was python script
ps aux | grep python
strace -p <pid_from_above>

This will show you all the system calls being executed. In my case it was SIGKILL which was killing the process. Actually god itself was executing it since the process was detaching from it and was trying to run as a daemon.

You can use strace for the following use cases:

• To check the system calls done by a command. This is helpful to know what all libraries the binary is trying to access.
  strace <command>
  Run strace echo hello for fun and check out the output.
• You can capture the output of strace to a file by passing -o flag and then use grep for analysis.
  strace -o output.txt ping 8.8.8.8
• Another use case for strace is when any of your application is running unexpectedly slow. Just pass -c flag to strace and you'll get statistics of all the system calls executed. You can also pass -p with -c to supply a pid.
  strace -c ping 8.8.8.8

I recommend you to read Solutions for tracing UNIX applications at IBM Developer Works for a more detailed tutorial.