This time, I thought I’d share some development experiences I’ve gained, while writing our RawSpeed raw file decoder.
When developing software, it is my general attitude, that applications should never crash. Fortunately many of you fellow developers agree with me on that issue, but there are still a lot of cases where we experience crashes. When reading an input, like a binary RAW file, there are a lot of things to take into account – also much more than you can deduce from reading specs and looking at your code.
So after writing the main bulk of the RawSpeed decoder, I decided to test the robustness, by doing “Fuzz Testing”. The basic premise is that you feed your decoder “junk” and see what happens. Naturally you cannot just feed it random data, since 99.999% of all these cases would not pass the first few tests. So what it does is:
This way you cover the “corrupted file” scenario. Using this method you are likely to discover a huge amount of bugs and issues – buffer overflows – race conditions – data ordering assumptions – overallocating memory – and other strange stuff that you might never have though of. I’ve found more than 20 potential crash bugs by using this method alone. Many of them require very specific changes and values to be changed, so you are sometimes very amazed with what pops up.
It is important that you corrupt your image in a predictable way, because it allows you to re-create the crashes you might get again, to make sure you eliminated the error. This is usually done by seeding the random number generator with the same number before starting to test an image.
The second fuzz test I use is a “truncated file” scenario. This is also a likely scenario, both from the user side, and from yours, where you do not check whether you are jumping outside the file. This is a bit simpler:
All of this is easier (and a lot faster) if you decode your material from memory, but there is nothing stopping you from generating these test cases as ordinary files, or pipe them into your application.
When all of this is said and done, fuzz testing is not the holy grail to stable software. It doesn’t test different hardware setups/platforms. It doesn’t test any of the logic, that is outside the scope of the input files, ie. code that doesn’t directly depend on the input. BUT it allows you to catch a lot of strange cases, and make sure your application doesn’t crash, but instead handles the error gracefully.