Difference between revisions of "Debugger"

Latest revision as of 14:03, 6 September 2013

Debugging Net-SNMP Applications

This page contains some tips on debugging Net-SNMP Applications. Often, when an application crashes, you may be asked to run the application under a debugger, or provide details from a core dump. This page presents a simple introduction to one debugger, GDB.

GDB / DDD

GDB is the GNU debugger, which is a terminal-based debugger. DDD is a graphical (GUI) front end to GDB.

Normal invocation

The simplest way to run an application under GDB is like so:

$ gdb /usr/local/sbin/snmpd
(gdb) run -f -Lo

The '-f' is necessary to prevent snmpd from forking into the background, and '-Lo' tells snmpd to send log messages to STDOUT (i.e. print them in the GDB console window).

Libtool invocation

Net-SNMP uses libtool while building it's applications. This means that sometimes the applications in the source/build directory are not actually binaries, but shell scripts. The libtool script does some magic so that the applications will run using the shared libraries in the build directory, instead of any libraries installed on the system. To run an application with GDB, you have to run GDB through libtool, like so:

$ ./libtool --mode=execute gdb agent/snmpd
(gdb) run -f -Lo

Getting a backtrace (aka stack trace)

Once you have got an application running under GDB, you can easily get a backtrace, which will display the sequence of functions that were called to arrive at the point where the debugger is currently stopped.

(gdb) bt
#0  0x10153d78 in init_snmp(type=0x101bb140 "snmp") at snmp_api.c:854
#1  0x1000524c in main(argc=3, argv=0x7fe04636) at snmpd.c:910

Loading a core file

If an application has crashed, often it will leave a core file. This core file can usually be loaded into GDB to get debugging information after the fact. Simply add the path to the core file after the path to the application when starting GDB.

$ gdb /usr/local/sbin/snmpd /tmp/core.70816

Often this will tell you why the application crashed (e.g. SIGSEGV, aka signal 11). You can then get a backtrace to send to one of the mailing lists for interpretation.

If testing snmpd (or another application) from within the source tree, you'll need to run gdb via libtool:

$ ./libtool --mode=execute gdb agent/snmpd /tmp/core.70816

Attaching to a running process

If your agent is running and you need to see what's going on inside of it (and you can't restart it directly within gdb itself), specify the process ID at the end to attach to it:

$ gdb /usr/local/sbin/snmpd 12345

After which you can get back traces, set breakpoints, etc.

Valgrind / Memory Leaks

To check for memory leaks, you can try Valgrind. Note, however, that Net-SNMP relies on the operating system to cleanup memory when an application exits, so there will be lots of un-released memory when the application exits. Please don't report this as a bug. We are only interested in memory leaks that grow over time.

If you don't want to read the Valgrind docs, here is a quick example:

valgrind --tool=memcheck --leak-check=full --show-reachable=yes snmpd -f -Lo

Note that since some memory leaks may be reported anyway just from initialization allocations that were never cleaned up, the best way to check for memory leaks in your mib-module code is to run valgrind twice. In the first run, perform a single snmpget or snmpwalk on your MIB implemenatation. The next time, perform 1000 of the same task. Then compare the two runs and if the memory usage is different for some section of code (by a factor of 1000) then you've definitely found a leak.

Dmalloc

Detecting memory errors in Net-SNMP is not only possible with Valgrind but also with Dmalloc. While Valgrind is more powerful than Dmalloc, Dmalloc is supported on more platforms than Valgrind (e.g. Solaris and Cygwin). Proceed as follows to analyze Net-SNMP with Dmalloc:

1. Install dmalloc.
2. Modify <dmalloc.h> and insert #undef strndup just before the declaration of strndup().
3. Configure Net-SNMP with the option --with-dmalloc=<dmalloc prefix>.
4. Build Net-SNMP.
5. Make sure that the dmalloc function has been defined in your shell. See also the dmalloc manual for the details.
6. Set dmalloc options, e.g.

   dmalloc high -p print-messages

7. Start Net-SNMP, e.g. as follows:

   agent/snmpd -f -Lo -Mmibs

Profiling

To look into performance issues, profiling helps a lot. Here is how to do it.

Using gprof

First, you need to build a static version of Net-SNMP with profiling information:

 ./configure --enable-static --disable-shared --with-cflags=-pg --with-ldflags=-pg ...

Then, run the agent and/or your commands as usual. Upon exit, they will leave a gmon.out file in the current directory. If you run multiple commands, do it from different directories so the gmon.out files don't overwrite each other. E.g.

 /path/to/snmpd -f -Lo --rocommunity=foobar localhost

Finally, run gprof to analyze the profile data:

 gprof /path/to/snmpd gmon.out

Using strace

An alternative program to gprof is to track the system calls using strace. gprof is a significantly better tool, but you may already have strace installed.

E.g. usage on Linux:

 strace -tt -p <pid_of_snmpd>

Other platforms may have a similar tool with a different name, e.g. truss on Solaris.

References

• Stack traces for Fedora RPM packages
• gdb mini intro