We need to able to reproduce

Good dev environment is key here. While it’s possible to do this on a production platform it’s highly discouraged for the following reasons:

Things that typically can cause problems in a program, and thus are required to be known in order to reproduce the problem are:

Any exception thrown in a program will have an error message. A lot of the time just reading the error message will give you the problem.

All programming languages have a stack. Each level in the stack will have a local scope with its current state. A new function/method call within the code will create a new level in the stack, with a new local state, where the code is executed. IIn most programming langauges this information is given back when an exception is thrown by the language. Reading the stack trace allows you to trace how the program came to the current invalid state in the current space and time of the program. It will also give you the file, line number and the error message of what caused the (immediate) error.

Should maybe be obvious, but this includes not just your own code, but also that of others; including libraries, frameworks and systems.

A debugger allows you to run a program and at will stop it at a breakpoint. Once the program has been stopped at the breakpoint you can inspect the current state and perform a number of operations. Notice that it is only one thread that has been stopped. Any other threads in your program are still running.

In addition the more advanced debuggers will allow you to go up in the call stack and inspect the state there as it was before you arrived at the current state.

Asserts can be used in what is called defensive programming, but also in the phase where you try to isolate infection chains. You test that the data has a certain shape at the beginning of a function/method before proceeding with calculations on the data. Takes the form of a boolean assertion (has to return true) along with data that is thrown if the assertion return false. The data typically is a string, depending on the language the assertion also allows you to send along other data that could be important. A well designed assertion will give enough data to be useful at the error message level.

Build a graph of state dependancies from the code base where each variable you think might be a candidate for an infection is mapped out in terms of what it effects, and what the variable itself effects.

Googling an error message (or looking at SO) and/or stack trace can often get you to the answer of what potential causes are. For common errors occurring in popular languages/frameworks/libraries this can significantly reduce the time to find a solution. There is a danger here however, and that is that it might have taken the original author significant time to come to the conclusion that he/she did, and you have skipped that entire process. Sometimes there is a significant surface area that this error can occurr in, and by copy and pasting a solution you have not really understood the problem itself and as a result most likely just made the problem worse.

Print is a very simple and easy measurement you can put in for some quick and dirty checks on how your code performs. While debuggers gives you a lot of power to slow things down, sometimes timing issues prevent you from using debuggers and there prints can be a saving grace.

There are several libraries for almost any programming language that supports logging. Logging is a standardnized way of logging information to a sentralized repository of logs. The log repository can be a server, files, database, etc. The easiest to implement is a standard log file where all information sent to the logger is gathered. Logging can have several levels of logging (DEBUG, TRACE, INFO, WARNING, ERROR, CRITICAL, ALL) that allows you turn on and off the verbosity of logs. Logging typically involves a bit of work in the beginning, but pays dividends ever after. Logging is also one of the techniques that can be used in production to gather important data from a running application.

Narrowing or flag planting is a technique where you use print/log statements to see how far your code executes before it crashes. The printing can also contain useful information of the current state between each executed statement. This can sometimes be considered a poor man’s debugger, but depending on the program it can be a very powerful technique. It typically shines when you start to have multiple threads involved that interacts tightly with each other. In situations like that a debugger typically struggles as it can only halt a single thread at a time.

For situations where you don’t really know where a bug might appear and the stack trace doesn’t really give much useful information you can use divide and conquer. Take a point in the program and diverge the path in that point. If the program no longer crashes the bug exists in the other half. Find a new point, rinse and repeat until you have honed in on the area where the bug occurs. Start applying other techniques at this point.

When firing in the dark there is no way to see how close you are to the target, and so every fifth to tenth bullet is a tracing bullet, which lights up the path it travels so that you can see where you are aiming. Tracing bullets in software follows the same idea, where hard to track down bugs are given a unique id which acts as a tracing bullet for data that travels through your system. At each point you can see how the data has changed over time. Used in combination with logging it can be used in production systems to trace how data travels through complex systems.

For advanced usage you can use tracing tools (language specific) to track how a variable changes over time. This can be a very spammy tool, so use with caution.

Git has a lot of history and lots of tools for digging through the history of the project. Seeing the changes to a function/method over time can give ideas of possible errors as well as give an idea of how it’s developed and so put things into its proper context.

When asking a friend you’re forced to unpack the problem so that he/she can understand the problem. This in itself can help you come across a solution. The questions asked by your friend as you explain also tend to help you find new angles on the problem.

Old story of a programmer who used to have a rubber duck at his desk. Whenever he ran into a problem he would sit down and explain the problem to the duck. If you lack any handy friend nearby, this is a good substitute.

Small piece of code that runs the problem code in a managed way.

Working with something that takes a lot of time to interact with, or is difficult to interact with, but you are sure is not the cause? Write a stub/mock that mimics the behaviour and have it replace that part in your code to increase the feedback loop.

Examples can include payment gateways, emails, anything I/O driven.

There’s a bunch of stuff you can do here that you really shouldn’t, but sometimes it’s what’s left. If possible, avoid using injections and directly editing library source code.