Debugging a CDC Crash on Linux with GDB

Why does CVM suspend when it receives a signal 11 (SIGSEGV)?

When a CDC process crashes on Linux, you usually will see a message like the following:

Process 1234 received signal 11, suspending

CVM (the VM used by CDC), installs signal handlers for most signals that are indicative of a crash, and suspends the process when the signal is recieved. This is in lieu of allowing a core dump. This has advantages and disadvantages. The disadvantage is that there is no core dump. The advantage is that you can still attach gdb and debug a live program. The following instructions will tell you how to instead produce a core dump if that is what you want, or how you can debug the live program.

How to make CVM core dump

If you want to make CVM produce a core dump rather than suspend the process that crashed, you need to do the following:

• In src/linux/javavm/runtime/sync_md.c, find linuxSyncInit() and remove the following from the signals[] array: SIGSEGV, SIGBUS, SIGILL, and SIGFPE.
• If you are building with the JIT:
  • You need to disable trap-based NullPointerExceptions (NPEs). In src/linux-<cpu>/javavm/include/jit/jit_arch.h, look for #define CVMJIT_TRAP_BASED_NULL_CHECKS and change it to a #undef. This will make it so generated JIT code does explicit NPE checks rather than rely on a SIGSEGV and and help from signal handler handleSegv().
  • Also in jit_arch.h, #undef CVMJIT_TRAP_BASED_GC_CHECKS and CVMJIT_PATCH_BASED_GC_CHECKS if they are present.
  • In src/linux-mips/javavm/runtime/segvhandler_arch.c, disable linuxSegvHandlerInit(). You can make it just return CVM_TRUE right at the start.

Now if you rebuild CDC and it crashes again, you should get a core dump that you can debug with GDB.

How to debug when CVM suspends a crashed process

After CVM suspends a crashed process, you can attach to it with gdb and start a live debugging session. However, this is a bit tricky since the process has been suspended. You end up needing two shells: one with the suspended process and one that you run gdb in. This is because you can't run gdb from the shell that also needs to resume the suspended process. Below is an example gdb session:

SHELL #1: Start the program and produce a crash.

$ bin/cvm -cp testclasses -Xjit:compile=all Test
<snip>
Process #17416 received signal 11
Process #17416 being suspended
[1]+  Stopped                 bin/cvm -cp testclasses -Xjit:compile=all Test

SHELL #2: Start GDB and attached to the suspended process

$ gdb bin/cvm 17416
GNU gdb 5.3
<snip>

SHELL #1: Bring the suspended process to the forground so it resumes execution.

$ fg

SHELL #2: Now that the program has resumed execution, gdb can finish attaching to it. The PC will be in the kill() funtion. You need to type continue, which will cause execution to resume at the crash point. This will cause the crash to repeat, but gdb will handle the signal this time instead of CVM.

Attaching to program: bin/cvm, process 17416
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
<snip>
0x4008acb4 in kill () from /lib/libc.so.6
(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x000ac404 in findSpillLoc (con=0x0, size=1, isRef=1) at ../../src/portlibs/jit/risc/jitregman.c:308
308         for ( spilloff = con->maxPhiSize; ;spilloff += 1){

At this point you can do a backtrace, or whatever other gdb debugging you care to.

Running CVM from within GDB

If you can reproduce a crash easily, rather than going through the above recommended code changes or debugging session, you can instead just run cvm from within GDB. It will handle the crash signal instead of CVM suspending the process. However, keep in mind that when using the JIT, there are two cases where these signals are part of normal operation and should be ignored (you can just type "continue" when you hit them)

• Trap-based NullPointerExceptions
• Trap-based GC Checks (only when CVM_MTASK=true or USE_JUMP=true)

The details for these trap-based features will not be covered here. What is important to know is that during normal program execution they intentionally result in a SIGSEGV, and the CVM signal handler performs the appropriate operation based on the current PC at the time of the SIGSEGV.

Hitting either of these while in a debugging session is rare, since you usually don't see NullPointerExceptions in properly running programs, and hitting GC CheckPoints that result in the signal is also pretty uncommon. However, if they are getting in your way, you can disable both as instructed above in the How to make CVM core dump section.

Build flags to help you debug

Once you've had a crash, you should try building with CVM_DEBUG=true. This will give you non-optimized code, gdb symbols, asserts, tracing support, and some debugging support functions you can call from gdb. Most of these are described in the CDC Runtime Guide and the debugging chapter in the CDC Porting Guide. If using CVM_DEBUG=true produces too slow or too big of an executable for you, you can also control these features individually. Below are some flags you can set on the make command line, and their default values:

In summary. CVM_DEBUG_ASSERTS=true, CVM_TRACE=true, and CVM_OPTIMZED=false are the flags that slow down performance of the VM. The other flags do not directly affect VM performance, but may do so indirectly by affecting one of these 3 flags (such as setting CVM_DEBUG=true).

Useful build flag combinations for debugging

The following build flag combinations are useful when debugging. They give varying amounts of debugging support, depending on how much of a performance hit you are willing to take. Flags that are not specified but may have non-obvious default values are included in parenthesis.

This build suffers no performance loss, but includes debugging symbols.

This build also suffers no performance loss, but includes debugging symbols and extra debugging functions that can be called from GDB (See Debugging chapter of the CDC Porting Guide).

This build will be slightly slower than a fully optimized build, probably about 10% to 20% on average. It adds assertion checks. It is especially useful during long running stress tests since it will catch bugs with the asserts, but not slow down the testing much.

Same performance as the previous set of flags, with assert checks being the only thing slowing the vm down. However, it includes all debugging support except for tracing, although the optimized code can be a bit hard to debug with.

If you want to be able to debug a crash that takes a while to reproduce, and you don't need tracing support, these build flags are a good choice. Not including tracing support will speed up the interpreter quite a bit.

-- ChrisPlummer - 09 Feb 2007