Testing in image processing – an open-source LabVIEW weekend project – Part 1

Image processing is hard. It’s fuzzy, it’s complicated, and the results are really hard to test.
When writing an image processing algorithm, you can have two kinds of errors
(1) “regular” bugs, like in normal software, and
(2) issues with performance, like: The algorithm doesn’t do what is supposed to do.

(1) is, well – not fixed, but at least somewhat known and best dealt with with things like TDD – unit tests, integration tests,..
(2) is more tricky. So far, my method is to hand-process two sets of sample data. Huge sets of sample data. (I need a new intern, boss!)  Then I use one half of the data for developing , and the other one for .. well, the finalisation of my algorithms.

The problem I’m walking you through has huge numbers of sample data. That’s not an issue at this time.

We’re operating at super high speeds here, which translates to 100s of pictures a second. But even if you are not doing crazy inline stuff, you should get a sample size that is as big and as representative as possible (which is a discussion for another time, but..).
If you understand your problem well enough, create edge case samples yourself.

The first step is highly customized and boring.
You need to do what the computer would do (One of my coworkers recommended me to “write a script” for sorting the pictures. Thanks, M., that’s exactly what I’m trying to do…) – and manually process the images.
I recommend you to write yourself a quick app that cycles through the images.
The information (in this case, there is either “feature at position x,y”, “ambivalent” or “no feature”) is stored in the file name, also, an unique identifier is added.

Here is a screenshot of such a quick and dirty program:

I did blur out some customer specific information (especially a description of how the file-names are generated), and this is not open-source, because I wrote it for work.

But since it’s not the first time I’m doing such things, I’d like to point your attention to two things

a) Give some status feedback
b) Provide the last filename(s)
At some point, you will be tired, lose focus, make mistakes, and want to correct them.

With this project, I have much (about 10 percent) data that is unclear – that is, I – as a human – can’t decide whether the feature is okay or not okay. So I don’t care what the algorithm does – and that reflects in my framework!
(With other projects, I’ve included categories such as “Must work” “Hard”.. in my testing framework, but I’ve found that this kind of categorization is better done in a second step. The brain is good at doing exactly one thing at the same time, and this requires you to switch.)

As a second step, divide the samples into 2 groups (Tests for development, tests for finalization/verification)

As a third step, you need to prepare your image processing algorithm.
Best way is to turn it into a dll, that  takes an absolute path of an image and saves it into a processed/samefilename.bmp or something like that. It should also return the file name.
Theoretically, you could use return values for results, but I’ve found that storing relevant information in file names is simpler and works for me.

As a fourth step, you need to actually test your algorithms. 

I’m gonna give you a framework in the next post, it  isn’t the most well-designed LV-code I’ve written, but I’m not playing architect on a weekend 🙂 Adapt it to your own needs (it does require Vision, but that can be easily replaced, remove dependency on BMP filetype),..


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: