Demos, Tests and Databases

This might be too much for one entry, but here goes. The art of testing usually involves some form of data and some configuration of machines, operating systems and other items that constitute the 'system'. Over the years, one of the biggest challenges is to recreate that actual environment as the customer would experience it. And it is a thorn in the side of most test professionals. How does it relate to quality software?

It relates to quality in terms of quality are a measure of customer perception, in terms of real dollars spent and in terms of the relationship between testers and developers that is so important to efficient scheduling.

1. Hardware costs money, so the same hardware used to test the software is sometimes shared with the people doing demos. Usually, but not always these people are marketing people. Consider when it is for beta testing using customer data and multiple customers see it during demos.
2. It is common for the data to become hopelessly out of date in the test area. So that even though you have the hardware you need, you don't have the real data. So, one of two things happens. You report bugs, which quickly get bounced back by the developers, because you were using bad data. Or, you don't find any performance issues, because your data set doesn't accurately reflect the size of the data set.

You may have heard that live data has been used to test passports - particularly presidential candidates information. This is actually quite common activity. Many times I will be told that live data, live systems (with test data) are used, especially for performance testing. What is it doing to the performance of the actual system?

Or, what happens during a demo when competitors get to see your customers data? Or, what happens to the test schedule when you can't get the system you need? Or it got hopelessly reconfigured -- again, with invalid bug problems.

There are solutions to this -- never share systems between departments, set up rigorous procedures for sharing systems, create transparency with your customers if you are going to use their data -- get their permission and allow them to audit your use of it.

What does quality mean to you?

This question has been asked in many times and many ways. Robert Pirsig famously wrote Zen and the Art of Motorcycle Maintenance and followed it up with another book called Lila whose main theme was the struggle of defining quality.

In software, the definition of quality is similarly elusive. I prefer the definition that says that you need to understand the core purpose of the software. And that can't break. This entry is about testing. There has been a lot written lately about how you can't test everything in today's software. There are lots of reasons for that. But that is for another discussion. Here, I want to say that all test planning needs to start from the core purpose. If you have software that controls a car wash, then first and foremost, the car has to be clean at the end. It can give you bad paperwork, and the customer may tolerate it, but the car must be washed.

In selecting tools, the size of the test team, and planning the rounds of testing, core is vital. I have designed some automated test suites, and each time, the first thing I wanted to write was core tests. So that, no matter how many changes happened to that software, core would still work. Because I believe that is what the customer needs and expects.