Fuzz Testing for File Input
This time, I thought I’d share some development experiences I’ve gained, while writing our RawSpeed raw file decoder.
When developing software, it is my general attitude, that applications should never crash. Fortunately many of you fellow developers agree with me on that issue, but there are still a lot of cases where we experience crashes. When reading an input, like a binary RAW file, there are a lot of things to take into account – also much more than you can deduce from reading specs and looking at your code.
So after writing the main bulk of the RawSpeed decoder, I decided to test the robustness, by doing “Fuzz Testing”. The basic premise is that you feed your decoder “junk” and see what happens. Naturally you cannot just feed it random data, since 99.999% of all these cases would not pass the first few tests. So what it does is:
- Read a valid RAW file into memory
- Copy the file data in memory
- Change 1000 randomly (but predictable) selected bytes to 1000 different (but predictable) random values
- Attempt to decode the image
- Hope it doesn’t crash
- Do this a couple of hundred times for all your images in your test library
This way you cover the “corrupted file” scenario. Using this method you are likely to discover a huge amount of bugs and issues – buffer overflows – race conditions – data ordering assumptions – overallocating memory – and other strange stuff that you might never have though of. I’ve found more than 20 potential crash bugs by using this method alone. Many of them require very specific changes and values to be changed, so you are sometimes very amazed with what pops up.
It is important that you corrupt your image in a predictable way, because it allows you to re-create the crashes you might get again, to make sure you eliminated the error. This is usually done by seeding the random number generator with the same number before starting to test an image.
The second fuzz test I use is a “truncated file” scenario. This is also a likely scenario, both from the user side, and from yours, where you do not check whether you are jumping outside the file. This is a bit simpler:
- Read a valid RAW file into memory
- Copy a random chunk (from the beginning) of the file into a new region of memory
- Attempt to decode the image
- (as above…)
All of this is easier (and a lot faster) if you decode your material from memory, but there is nothing stopping you from generating these test cases as ordinary files, or pipe them into your application.
When all of this is said and done, fuzz testing is not the holy grail to stable software. It doesn’t test different hardware setups/platforms. It doesn’t test any of the logic, that is outside the scope of the input files, ie. code that doesn’t directly depend on the input. BUT it allows you to catch a lot of strange cases, and make sure your application doesn’t crash, but instead handles the error gracefully.
12 responses to “Fuzz Testing for File Input”
Leave a Reply
if You need more crashing input i can send You some panasonic lx5-rw2-files which crash rawstudio-daily ;)
Cool :)
Actually I just uncovered a crash issue with Panasonic images, this weekend, while fuzz testing (fixed in rawspeed rev 296, 4 days ago). It didn’t take corrupted images to crash though ;)
If crashes on the latest daily, please enter a bug, or write on the dev mailing list.
As a sysadmin, I can’t help but cringe at your statement that software should never crash. Crashing seems such an honest admission of failure compared to, say, corrupting my precious data. :-) I admit, though, that the tools of a sysadmin are rather different beasts from typical end-user software like Rawstudio.
To me it sounds like most of this would be covered by the kind of boundary condition tests you’d normally include as the most natural thing in your unit tests. That being said, monkey testing certainly has its place, not least because it’s much easier to get the computer to generate test cases than it is writing them yourself.
Your point about repeatability is very important. Without repeatability the test will be merely a source of frustration, only demonstrating that a bug exists but being no help in finding it. Generally speaking, random tests are great for finding the unexpected, but they should never be relied upon to test for what we can predict. More valuable than any amount of testing is a healthy dose of paranoia towards any kind of input. Again speaking as a sysadmin, I’ve found it safe to assume that the world is full of the malicious and the incompetent, and both love nothing more than throwing data at your code. :-)
>>As a sysadmin, I can’t help but cringe at your statement that software should never crash.
Sorry – I don’t get your point at all. In my mind, crashing is exactly the opposite of what you describe. Instead of gracefully handling error conditions, you just shut down, leaving all of your data in limbo – what if the crash happens when this or another thread is writing data – *bam* data gone or corrupted.
I honestly cannot see a single case where crashing the application is better than handling the errors gracefully.
>>…you’d normally include as the most natural thing in your unit tests
The point here is that this testing uncovers cases you would never have identified as cases for a unit test. When you do unit tests, you test for bugs you already know you have covered. You can of course do unit tests for all the issues you discover.
>> I honestly cannot see a single case where crashing the application is better than handling the errors gracefully.
That wasn’t my point, so maybe I was being unclear. Failing well is supremely important, and in most cases, crashing will be preferable to continuing with an unhandled error and possibly data corruption. Graceful (and correct) handling of an error is mostly preferable, though. (The ‘mostly’ qualifier because there are always so many things that can go wrong that you often don’t realistically need to deal with, and handling them all will just mean that 90%+ of your code will be error handling.)
We generally design our code so that failures will not be catastrophic. For instance, we design commands to always be safe to run again in case of failure (i.e. ‘do only what’s needed to set the world right’), we defer writing/committing data to the end of a job where nothing will be half-done, we have various automated sanity checks (if the next update is going to affect more than 5% of a database table, don’t do it without confirming) and so on. Failures will necessarily happen, so we build software that can mostly deal with it.
Why didn’t they teach this shit in CS? :-) We learned plenty about testing and code contracts, but not much about how to minimise the effect of failures. Well, programming as a sysadmin is rather different from everything else I’ve ever coded, so maybe that has something to do with it.
>> The point here is that this testing uncovers cases you would never have identified as cases for a unit test. When you do unit tests, you test for bugs you already know you have covered. You can of course do unit tests for all the issues you discover.
Unit testing is not just for input that you know will trigger existing or fixed bugs, but also for prodding classical weak spots like boundary conditions. That’s where most bugs that aren’t brain farts occur in my experience. Even if no problem exists near a boundary today, an innocent or thoughtless change elsewhere can easily introduce one tomorrow. There’s no reason not to do simple regression testing in one’s unit tests. It’s also useful to keep pushing those buttons to check that the code fails like it should. If code works when it shouldn’t, that’s as much cause for alarm as when it fails when it shouldn’t.
You should probably do feedback-directed fuzz testing too.
http://scholar.google.com/scholar?hl=en&q=feedback-directed+fuzz+testing
Also, you didn’t mention which fuzzer you are using. You might want to try zzuf.
Rawstudio don’t run with a computer without SSE (Athlon Tunderbird 1.4). Program crash when i open image.
@Bruno: Please keep the topic here. Please file a bug report, preferably with a backtrace of the crash location. Thanks!
This?
[………. EDITED BY KLAUS POST……..]
RawSpeed Open /home/bruno/Immagini/STUDIO CASALINGO/_DSC2997.NEF: 0.289s
RawSpeed Decode /home/bruno/Immagini/STUDIO CASALINGO/_DSC2997.NEF: 1.614s
Istruzione illecita
“
I removed some of the text – please go here to help me:
http://bugzilla.rawstudio.org/show_bug.cgi?id=454
Thanks
Do you already have a planned release date for a public beta or the final product?
I don’t know if my files are too big for raw-studio daily (Sony a850 files – 35 mb) but it crashes when I browse quickly between files. Any idea for a workaround?
Normand Desjardins
desjardins.normand at gmail.com