A new process was developed for selecting records from a large database. It was carefully tested and seemed to be doing precisely what it was supposed to do. The real test is to compare the results with those from the existing process to find out if there are problems. This was done by the user and they reported that they thought the counts were way off.
Part of the selection process was to catagorize the records selected. When some catagories came up with no records found, it was clear that there was a problem with the new selection process. A correction was made and a rerun of the selection yielded very similar but different counts.
The new process gave slightly higher counts in most categories, lower in others. Part of this was correctly attributed to the time frame involved: part of the criteria was based on activity in the previous 12 months — the selection in the new process was run later so that there was some new activity, but also records where the activity was just over 12 months old would not have been selected.
To find out the cause of any other differences, the two different selection files were compared. It was found that there were about 1000 (out of 35000) records selected by the new process that had not been selected by the old process. There were also about 900 records selected by the old process that weren’t selected by the new process.
Analysis of these records uncovered the fact that the old process was selecting records that were supposed to be skipped and not selecting other records that were supposed to be selected! It also uncovered about a dozen records that the new process missed selecting.
The moral of this little story is that just because the new doesn’t match the old that the new must be wrong. But there is more! It also suggests that it is wise to audit existing processes from time to time. In this case, it was clear that most of the selection criteria were being handled properly — it was more recently added criteria that wasn’t being correctly handled.
Even if you have no plans to replace a data selection process, perhaps now would be a good time to schedule an independent audit to verify that you are getting what you think you should be getting!