Lightweight debugging on embedded Linux

Question

I'm developing an application that runs on a small Linux-based SBC (~32MB RAM). Sadly, my app recently became too large to run under GDB anymore. Does anyone know of any good, lightweight debugging methods that I can use in embedded Linux? Even being able to view a thread's stack trace would be extremely helpful.

I should mention that this application is written in C++ and runs multiple threads, so gdbserver is a no-go as it doesn't work with multithreaded apps.

Thanks in advance,

Maha

Answer 1

gdbserver definitely works with multi-threaded applications, I'm working on an embedded project right now with >25 threads and we use gdbserver all the time.

info threads 

lists all the threads in the system

thread <thread number from info threads>  

switches to that thread of execution.

thread apply XXX <command>  

Runs on the thread designated by XXX, which can also be 'all'. So if you want the back trace from all running threads do

thread apply all bt

Once you're in the execution flow of a given threads all your typical commands work as they would in a single threaded process.

Answer 2

I've heard of people doing hacks like running the application in an emulator like QEMU and then running GDB (or things like valgrind) on that. It sounds painful, but if it works....

Would you get anywhere with libunwind (to get stack traces) and printf-style logging?

Answer 3

Serial port printing is the most light weight I can think of ~~~ Easily seen in a Host PC, and simple and light weight code inside your app~~

If you do not have a serial port, once we used an GPIO port and simulated a serial port using it. It worked perfectly well, but was a bit slow :-( ~~~

Answer 4

Is there a reason why you have built your own debugger? I am developing a Linux system using an ARM processor (AT91SAM926x) and we are using both compiler and debugger from CodeSourcery. I do not think that they have released a version with GDB 7 yet but I am debugging multithreaded C++ applications using the gdbserver tool without any problems.

Answer 5

Gdbserver does indeed work with multithreaded applications. However you do need to compile a cross target debugger for your host to get it to work with your target gdb.

See this article for a detailed description of how to do it:

Remote cross-target debugging with GDB and GDBserver