Debugging Techniques

If you write perfect code that runs the first time (and every time), move along—there’s nothing to see here. These are not the droids you’re looking for. For the rest of us, who spend at least some of our time debugging embedded code, I’ll offer some of the debugging techniques I’ve found effective over the years.

First, some history. When I first started doing embedded development in the 1980s, development tools were primitive compared to what we have today. Yes, there were professional-level development systems, like the Intel Intellec 8 for the 8080A, but these systems tended to be expensive and small companies (like the ones I worked for in this era) couldn’t afford them.

The next step in the evolution of embedded debugging was the ICE, or In-Circuit Emulator. Early ICEs were big, standalone units made by companies like Hewlett-Packard and later units were interfaces between a PC and an embedded system. They consisted of a hardware interface that typically used a bond-out version of the embedded microprocessor. These were special microprocessors with debug and bus signals brought out to unused pins and were used by the ICE to control the micro in the target circuit. Installing one involved removing the micro from its socket in the test system and replacing it with the bond-out micro, which was in turn connected to the ICE via a ribbon cable.  ICEs took complete total control of the target micro and supported loading code, breakpoints, single-stepping, and displaying CPU registers and memory.

My one and only experience with a true ICE was an HP-64243A for the Motorola 68000. List price back in the day (circa 1984) was around $25,000. My company couldn’t that, so they rented one for a few months while I debugged the system (an SMD disk controller for the VMEbus). Although HP had a C compiler for that beast (for money, of course), I did all of my work for that project in 68K assembly.

Modern microcontrollers integrate most of the functionality of an ICE directly on the die, so it’s no longer necessary to swap the part out for a bond-out version. All of the debug circuitry is built into the micro and uses four pins to talk to the outside world over a JTAG interface. The JTAG interface is a small gadget with a cable that connects to a JTAG header on the development board and a USB cable that connects to a PC.

There are many proprietary, third-party, and open source versions of the JTAG hardware and interface software available. Too many, in fact, to go into details here—perhaps I’ll do a blog post on this subject in the future. To cut to the chase, you need a target adapter and interface software compatible with it. Most commercial JTAG target adapters provide interface software with their hardware that works with most of the popular development tools (Keil, IAR, Eclipse, etc). Some microcontroller vendors (such as Infineon, ST, and NXP) sell their own low-cost JTAG adapters that usually work only with their own products. The cost is so low, that these are good options (compare an ST ST-LINK/V2 at $20 with a Segger J-LINK Base at $400) if money is tight (and it shouldn’t be—that’s the topic for a future post).

Hardware debugging, of course, is not the only way to debug embedded code. Many people to this day still sprinkle printf() statements at strategic places in the code. This time-honored practice is a viable technique, but it has several drawbacks, including the need to add the printf() calls to the code (and remove or disable them later). It also affects the timing of the code (particularly if the serial drivers are polled) and, worst case, changes the behavior of the code. I’ll only use this technique if I don’t have other, non-intrusive, debug tools available.

At this point I’m going to assume that most of you are familiar with basic debugging techniques like setting breakpoints and single-stepping code and skip directly to some of the perhaps not as well-known tools and techniques that I use.

Oscilloscope – An oscilloscope can be used in several obvious and not so obvious ways to debug embedded software. You can use one to look at signal integrity on peripheral bus lines (such as I2C and SPI)—are those signals close approximations to digital signals, or do they have lots of ringing and/or rise-time issues? A scope will tell you. Some scopes, particularly modern digital scopes, may even have built-in protocol analysis capabilities, which can make short work of debugging peripheral driver code. Scopes can also be used to measure timing deep inside your code. The basic technique here is to toggle an I/O output port at the appropriate place in your code and watch for the signal on a scope. Turn on an output port at the beginning of a function or ISR and turn it off at the end and you can determine the timing by measuring how long the signal is high on the scope’s screen. Toggling an I/O port is a much less intrusive operation than sending data over a serial port. You can get as elaborate as you want here, as long as you have enough available I/O pins. On one project I needed to see the current state of a state machine in real-time and I just happened to have three free pins available. I encoded the state on the three pins and used an oscilloscope to monitor it.

Logic Analyzer – If an oscilloscope is useful for embedded debugging, a logic analyzer is even more useful. If I could have only a scope or only a logic analyzer, 99% of the time I’d take the logic analyzer. Although you can’t see analog signal integrity issues on a logic analyzer, you can see many more signals simultaneously—most logic analyzers have 8, 16, or even more channels where scopes have only 2 or 4. Logic analyzers really come into their own when debugging serial protocols (I2C, SPI, CAN, serial, etc.) as protocol analysis is available on just about any analyzer you’re likely to see. Logic analyzers used to cost big bucks, but in recent years analyzers have become available in the sub-$1000 price range from companies like Saleae and Zeroplus. Saleae analyzers are wildly popular, and for good reason—they offer tremendous bang-for-the-buck. They’re not perfect, but they do what perhaps 90% of embedded developers need to do 90% of the time.

Data Breakpoints – Everyone is familiar with instruction breakpoints, but data breakpoints seem to be less familiar. If you don’t use them, and your microcontroller supports them, you should. Data breakpoints let you set a breakpoint on read or write access to a specific data address in memory. Something overwriting a variable or part of a buffer and you can’t figure where it’s happening? Set a data write breakpoint on that address and the debugger will stop when that location is written to. Some microcontrollers even let you specify the value of the data written to an address to break on (e.g. break when 0x1234 is written to address 0x4E00280C).

Disassembly View – Many debuggers let you step through code at the assembly language level. This can give you insights into behavior not readily apparent when stepping through C/C++ code. Don’t know assembly language? Learn it! Even if you don’t write code in assembly It’ll give you an advantage when debugging over those who don’t. Although each microcontroller architecture is different, once you learn assembly language for one microcontroller family applying that knowledge to other families isn’t difficult.

Fault Analysis – Most 32-bit microcontrollers have an extensive fault mechanism to handle things like bus errors, usage faults, and memory management faults. Learning how this mechanism works and how to interpret the breadcrumbs it leaves behind when a fault occurs can help you solve some of the trickiest issues. Some debuggers have built-in support for interpreting faults, but many do not. In any case, it’s wise to learn at least the rudiments of your micro’s fault mechanism.

Step Back – Take a break. Go for a walk. Sleep on it. Sometimes you’re banging your head trying to get to the bottom of a tricky bug and you find yourself getting nowhere. Now’s the time to go do something else and come back a few hours later (or the next day) and attack the problem with a fresh mindset.

Pair Debugging – You’ve heard of pair programming? This is the same thing applied to debugging. It’s particularly effective when you’re stuck and can’t seem to make progress debugging a problem. Get a colleague to come over and look over your shoulder as you go through the process of debugging the problem. Often, they’ll spot something you missed or have encountered a similar issue themselves.

Read Your Code – This one’s so simple I’m surprised more people don’t use it. Read through your code, concentrating on areas likely to be associated with the bug. Run the code through your mental “execution engine” and try to envision a sequence of inputs and events that can lead up to the error. Sometimes just thinking about an issue is more powerful than all other debugging techniques combined.

Logging Framework – Include a logging framework in your project to log debugging information to an output stream (serial, CAN, SPI), a storage device (like an SD card), or even to a circular buffer in RAM. You can write a logger yourself or use one of the many open-source loggers.

Now for some debugging “gotchas”:

Chip Errata – Working on a driver for a peripheral on a microcontroller and it’s just not behaving like the reference manual says it should? Perhaps it’s not your code, but a silicon bug. Run, don’t walk, to the manufacturer’s web site and download and read the errata for that part. If it is listed in the errata, you have several options: implement a workaround (often described in the errata itself), use a different, but similar peripheral on the chip, check whether a new stepping of the chip is available that corrects the errata, or, worst case, switch to a different part. I wouldn’t hold my breath waiting for the errata to be fixed, however, as many chip vendors take years to fix bugs (if ever).

Compiler Bugs – Check with the maker of your compiler if there are any known bugs likely to cause the symptoms you’re seeing, and if there’s a fix or workaround. Be aware, however, that compiler bugs are rare—it’s much more likely the issue you’re seeing is the result of a problem in your code rather than a compiler bug. Complain too loudly and too often about mysterious compiler bugs no one else has encountered and you’re likely to get a bad reputation among your co-workers.

Compiler Optimizations – Debugging code compiled with optimizations turned on is often frustrating. The debugger seems to jump around in the code seemingly at random and some code appears not to exist at all! This is a common issue with optimizing compilers. Turn compiler optimizations off, at least while you’re debugging. It’s generally okay to leave optimizations turned on for files you’re not debugging.

Interactions with the Debugger – Sometimes code behaves differently when run in a debugger than it does without the debugger. The often happens when peripheral registers are involved. This is best explained by an example: I was recently debugging some code that initialized the position interface peripheral on an Infineon microcontroller. This peripheral has registers for the current and expected Hall patterns and the process for setting these registers is to write the desired patterns to shadow registers and then set a bit in another register to initiate a transfer of the patterns from the shadow registers to the actual pattern registers. When I single-stepped this code in the debugger the patterns I wrote to the shadow registers were not getting copied to the pattern registers. When I ran the code without single-stepping it, the shadow transfers occurred as expected. The issue here is probably related to the debugger reading these peripheral registers after each instruction step. Some peripheral registers are sensitive to this type of read activity (by a debugger).

Hello world!

Welcome! This is a blog devoted to embedded systems topics, from the perspective my having spent over thirty years in the trenches doing embedded development and managing others doing it. I hope to share my insights with others less experienced in the field and to engage in discussions with those of you who’ve been around the block yourselves. I promise not to pull any punches and to tell it like it is, but keep in mind that most of what I offer here are my opinions–yes, opinions based on decades of experience, but opinions nevertheless.

Some of you may remember an earlier blog I wrote on similar topics. Rather than rehash that material, I’m starting again from scratch from a fresher perspective.

So what kind of topics will you see here? Mainly my experiences with tools, techniques, processes, and, yes, even the wide variety of personalities I’ve encountered along the way (but I won’t be naming names…) Some of the things I’ll cover are programming languages, debuggers and debugging tools, interfacing to hardware, development methodologies, and microcontrollers (and the companies that make them.)

In the next installment, I’ll discuss a topic near and dear to my heart: debugging techniques.