Debugger
Contents
Debugging Net-SNMP Applications
This page contains some tips on debugging Net-SNMP Applications. Often, when an application crashes, you may be asked to run the application under a debugger, or provide details from a core dump. This page presents a simple introduction to one debugger, GDB.
GDB / DDD
GDB is the GNU debugger, which is a terminal-based debugger. DDD is a graphical (GUI) front end to GDB.
Normal invocation
The simplest way to run an application under GDB is like so:
$ gdb /usr/local/sbin/snmpd (gdb) run -f -Lo
The '-f' is necessary to prevent snmpd from forking into the background, and '-Lo' tells snmpd to send log messages to STDOUT (i.e. print them in the GDB console window).
Libtool invocation
Net-SNMP uses libtool while building it's applications. This means that sometimes the applications in the source/build directory are not actually binaries, but shell scripts. The libtool script does some magic so that the applications will run using the shared libraries in the build directory, instead of any libraries installed on the system. To run an application with GDB, you have to run GDB through libtool, like so:
$ ./libtool gdb agent/snmpd (gdb) run -f -Lo
Getting a backtrace (aka stack trace)
Once you have got an application running under GDB, you can easily get a backtrace, which will display the sequence of functions that were called to arrive at the point where the debugger is currently stopped.
(gdb) bt #0 0x10153d78 in init_snmp(type=0x101bb140 "snmp") at snmp_api.c:854 #1 0x1000524c in main(argc=3, argv=0x7fe04636) at snmpd.c:910
Loading a core file
If an application has crashed, often it will leave a core file. This core file can usually be loaded into GDB to get debugging information after the fact. Simply add the path to the core file after the path to the application when starting GDB.
$ gdb /usr/local/sbin/snmpd /tmp/core.70816
Often this will tell you why the application crashed (e.g. SIGSEGV, aka signal 11). You can then get a backtrace to send to one of the mailing lists for interpretation.
If testing snmpd (or another application) from within the source tree, you'll need to run gdb via libtool:
$ ./libtool gdb agent/snmpd /tmp/core.70816
Valgrind / Memory Leaks
To check for memory leaks, you can try valgrind. Note, however, that Net-SNMP relies on the operating system to cleanup memory when an application exits, so there will be lots of un-released memory when the applications exist. Please don't report this as a bug. We are only interested in memory leaks that grow over time.
If you don't want to read the valgrind docs, here is a quick example:
valgrind --tool=memcheck --leak-check=full --show-reachable=yes snmpd -f -Lo
Note that since some memory leaks may be reported anyway just from initialization allocations that were never cleaned up, the best way to check for memory leaks in your mib-module code is to run valgrind twice. In the first run, perform a single snmpget or snmpwalk on your MIB implemenatation. The next time, perform 1000 of the same task. Then compare the two runs and if the memory usage is different for some section of code (by a factor of 1000) then you've definitely found a leak.
Profiling
To look into performance issues, profiling helps a lot. Here is how to do it.
Using gprof
First, you need to build a static version of Net-SNMP with profiling information:
./configure --enable-static --disable-shared --with-cflags=-pg --with-ldflags=-pg ...
Then, run the agent and/or your commands as usual. Upon exit, they will leave a gmon.out file in the current directory. If you run multiple commands, do it from different directories so the gmon.out files don't overwrite each other. E.g.
/path/to/snmpd -f -Lo --rocommunity=foobar localhost
Finally, run gprof to analyze the profile data:
gprof /path/to/snmpd gmon.out
Using strace
An alternative program to gprof is to track the system calls using strace. gprof is a significantly better tool, but you may already have strace installed.
E.g. usage on Linux:
strace -tt -p <pid_of_snmpd>
Other platforms may have a similar tool with a different name, e.g. truss on Solaris.
