Merging modern software development with electrons and metal
Random header image... Refresh for more!

Regression Testing with Cognex Insight Smart Cameras

Regression testing tries to verify that software changes do not cause current functionality to fail. A few years ago I wrote software to do regression testing for jobs on a Cognex Insight vision system. Since I do not have a Cognex system at my desk, I cannot give all the juicy details, but I can give an outline.

Machine builders often build a machine to work on a small set of sample parts (say 50). But when it gets into production with many more parts, there are often problems because the production parts show wider variations than the samples (or the manufacturer has made changes).

It’s the same for machine vision – you have to pick one particular part to start designing your machine vision job. Suppose the job works well in production, but somebody wants better results – so you grab a new part and get to work, right? Well, if you’re not careful, you can end up with a vision job that works great for that new part, but not for a typical part, and thus end up worse than you started.

So what I did was save a whole bunch of pictures of good parts and bad parts from production runs, then after I made any changes to the vision job, I ran my part database through the camera, and checked the results.

The Insight cameras use some sort of PowerPC processor, have Ethernet, RS-232 serial, and a bit of digital I/O (e.g. for trigger input). The Insight Explorer user interface software runs on a PC, is written in Java (and works pretty well; note that some newer Cognex products use the .NET framework), and uses a spreadsheet approach to machine vision, which has its pluses (such as simplicity) and minuses (like trying to sequence actions).

I like having the cameras on the network; you can work at your desk with the camera mounted far away. Cognex makes it especially nice by using standard Internet protocols, such as ftp to load and save jobs and pictures (BTW, if you need a free Windows ftp server or client, you should look at FileZilla), and telnet to control the camera.

Using telnet is basically like having a command line interface to the camera. You can do a lot with the camera (load jobs, trigger the camera, insert data, get results, etc), and it’s easy to test out ideas at the telnet command line, then codify them into a program.

I used Python with a free, open source telnet library. At least at the time, there was no free telnet library for .NET, and it didn’t make sense to buy one for this simple application. Then I wrote a Python module to do all the camera control I needed.

To do the regression testing, I wrote a Python program that loaded the desired vision job, then went through the database of pictures, loading each picture, triggering the camera (so it would run the job on the loaded picture), recording the result, comparing the result to the desired result, and then scoring the overall results.

Theoretically, it would be possible to do the same thing using the Insight Explorer software without using a real camera. In that case, you would use a GUI functional testing library (and some good free ones exist for Python) to load the jobs and pictures, then check the results. However, since Insight Explorer is written in Java, the normal GUI testing tools did not work (Java visual widgets aren’t the same as the native Windows ones – OK, maybe if you’re using SWT, but I think they were using Swing). I had limited success automatically inserting keystrokes and waiting, but it wasn’t reliable.

Tony

5 comments

1 Kevin { 02.20.08 at 7:27 am }

Hi Tony,

I’m trying to use telnetlib to talk to an Insight camera, and I’m not having any luck — read_until just times out returning no data. Any clues on how to proceed? It looks like I can log in, but I get no further.

I see you’re an Evelyn Waugh fan. One of my favourites of his is an oddball book called The Ordeal of Gilbert Pinfold. Semi-autobiographical account of a guy suffering from psychotic delusions, apparently based on an episode in Waugh’s life. Brideshead Revisited it ain’t.

Thanks

Kevin

2 admin { 02.21.08 at 1:26 pm }

Kevin,

The Cognex telnet code was a few years ago, but here are some pointers. I first used a normal telnet program (Windows telnet back then; now I’d probably use puTTY) to understand how the Insight camera worked, then automated the process using telnetlib.

If you can connect manually, you know the Insight telnet is working, that you have the correct user name and password, etc.

I did use a lot of read_until (and some read_eager). For example, to start the log on process, I would read_until the Insight’s telnet sent “User: “, then used write to send the user name (followed by the termination sequence “\r\n”). Termination sequences can be very important – right now I’m dealing with a serial motion controller that’s very picky about that.

Good luck,
–Tony

3 admin { 02.21.08 at 1:38 pm }

I should note that Brideshead isn’t my favorite Waugh novel – if I had to choose, I’d probably pick Vile Bodies, but it’d be a hard choice.

–Tony

4 Leslie { 03.07.09 at 2:14 am }

Hi,

Interesting and very relevant point re: regression testing. Definately something we all should do but often forget when in the thick of it on the factory floor sometimes.

However, if you use the latest version of Cognex InSight Explorer (approx. ver. 4.2 as of March 2009), you’ll find that the explorer GUI has all the functions necesary for full regression testing. Using the Record\Playback function, you can automatically load and inspect a sample set of images that were previously saved from actual production samples. Then, either with camera connected or simply using the software emulator, you can inspect the images and perform all the usual vision functions. It’s a really handy system for regression testing and general application development.

Leslie

5 Tony { 03.09.09 at 12:26 pm }

Leslie,
I did the regression testing back when Insight Explorer was at 2.70. One of the cameras was, IIRC, limited to 2.70 and couldn’t use newer versions.

The Record\Playback function sounds interesting. However, to make it truly useful you have to sort the images into pass/fail bins (directories?), and then make sure that the results were recorded, so you could automatically see the performance (false positives, false negatives).

I’m sure this could be done. One approach would be to use persistent counters to keep track of pass/fail results, but I would look at sending the results out over the network (with the image name if possible) so it would be easy to look at all the “wrong” results (fails that should pass & vice versa).

Leave a Comment