Ways and Means

I recently learned that New Jersey and Pennsylvania are putting in new toll benefits for seniors. Good, you think? Well, these benefits are changing form. The benefits will now be tied to EasyPass, which not all seniors use.

This reminds me, for some reason, of when Digital Equipment Corporation was trying to get people to stop using the PDP-11. Try as they might, people really liked the workhorse, and didn't want to change. Of course, this meant that DEC was selling less hardware than they would have liked, and spending a fair amount of their customer service resources supporting 'old technology'.

There are ways and means of motivating people to move to a new system. Usually, the young among us is more open to these changes. They haven't gotten used to the old ways. We move to new things, too, when they seem significantly easier than our old ways.

For the tolls in these mid-Atlantic states, the means to move the seniors to this new program seems cumbersome. Even if they already have EasyPass, they can't simply add a senior option. If they have been using coupons, they are forced to apply for EasyPass, even if they are skeptical of using electronic systems for tolls. It would be like telling people we no longer accepts checks.

When times comes to ask users to make a change, always consider the two most successful paths: new users and the new system being easier than the old. Otherwise, you'll find yourself in the situation that DEC did - new product, loyal customers, but not enough profit on the new product.

Flexibility in Data Definition

I was reading a book recently called "The Art of SQL" that I highly recommend. The authors warn against what they call the dangers of excess flexibility. It got me to thinking about automated test designers who promote 'data driven design'. In fact, if you say that you use data driven design, you're sure to be considered a serious Quality Automation Professional. But what does that mean?

Well, I have discovered a disturbing trend. The trend is to create a table of attributes, and corresponding values. The automated designer creates his or her engine, and all the users of that engine have to do is populate the table. That was the part that always got me. These professionals don't have to actually understand the product that was going to be tested, but they get paid for 'creating the engine'. It seemed that there was still an awful lot of work to do to populate these tables, but until I read the clear and concise description of excess flexibility, I couldn't seem to put my finger on just why it sounded so wrong.



Now, I do create data driven automated tests. These are created after I have analyzed how the product is meant to work. To use "The Art of SQL"'s term, I atomize the data. "The Art of SQL" defines atomized data as what is unique. By extending the principles of creating maintainable databases to creating maintainable data driven automated tests, you can create an automated suite that will actually survive past one Quality Assurance Test Engineer. I know. Several of the test suites I have created over the years are still in use long after I've left.



I used to try to figure out why the way I create data driven testing wasn't the way that they do it. Why do they 'get paid the big bucks', and how do I explain that my way is better?

Well, I don't have to anymore. I can just point you to "The Art of SQL". After all, SQL is all about managing data, and if you are going to create automated tests that are 'data driven', you should effectively manage the data.

Cross Training

There are different methods of cross training, and I'd be interested to know if others agree with my version.

The premise of cross training is that people can jump in at any moment to help out. It can reduce the need for redundant staff. It can relieve pressure when one project gets behind.

And I'm all for it.

In my version of cross training, there is a basic expertise that each person has, with declining expertise on other projects. Think of the skills checklist that you sometimes have to fill out for job applications. You create a scorecard for everyone. This allows you to identify gaps in training and plan for future cross training. And it becomes very valuable during scheduling.

With cross training, there must be an element of training, which is why when scheduling and planning cross training, someone must be training and someone must be learning. They must be on the same project together. Now, training can have several definitions. In this case, the person learning should be spending time observing and asking questions of the expert as he or she performs the job. That person then has specific tasks to perform on the project. In turn, the expert doing the training needs to be able to fill out the scorecard at the end of the project, observing how much has been learned and how valuable the person will be next time.

Finally, to ensure success, there must be some non-overlap in projects. If this cannot happen, then you can't really shift resources between projects without jeopardizing the schedule or the training.

Applying Standards in Small Organizations

I am a proponent of using good process when developing projects. I am NOT a proponent of process for process' sake, however. So, when I've had the opportunity to work in smaller groups, I am confronted with decisions on how much process is feasible. Usually, when I mention a standard, I get one of two reactions. Either someone is pro-Six Sigma, CMMi, or another specific standard, and wants to apply it all now. Or, I hear that there is nothing in the budget, and anyway, process would slow us down. But so many startups run like the wind to develop products, only to find things in a big mess as they grow. It would be more efficient to put a bit of process in at the start. What to do?

Well, ISO has acknowledged this and is working with a number of other standards bodies to develop standards for what they call "Very Small Enterprises (VSEs)". The name of the group is ISO/IEC NP TR 29110. They are looking at what Latin America has done.

In Latin America, they are already using a model called MoProSoft for certification that looks promising. In the United States, any organization under 50 is considered a small enterprise, but in Latin America, it is more typical to find companies from 1-10 - the definition for VSEs. Because most of these companies were working with governmental bodies, they needed to prove certification, in much the same way as in the United States - where government contract companies are typically bigger. Having gotten the jump on creating these smaller processes, they can be used as a model for other VSEs interested in implementing process.

Up until now, I have found that in order to use standards in small enterprises, it was necessary to take parts of standards and use them as guidelines. This has worked well, and turns CMMi a bit on its head. Instead of starting with an organization of Level zero, and improve, you lay the groundwork using building blocks of known standards. Both CMMi and Six Sigma devotees assume that in the beginning there was no process. But there can be. Simple processes can be implemented just to be able to recreate a build, or track the testing. Utilize simple analyses of the last release to learn from it. By putting building blocks in place in the beginning, transition is easier as you grow.

So I will be watching to see if the new standards for VSEs can be adapted to startups that may grow. If so, we will be able to apply VSE standards for smaller organizations. If the organizations grow, the transition to standards for larger organizations will become easier.

Follow up on Technical Depth

I've just read a great article from Gartner Resources on hiring the right IT personnel. It speaks of the dangers that hiring managers encounter when they focus on technical expertise without considering the abilities of the individual's ability to learn, ramp up and contribute to the overall business.

My first entry regarded how to discern technical depth. Gartner addresses how important technical depth should be when hiring. When you are examining a resume and interviewing a candidate, look for their accomplishments, especially as it relates to their previous job. How did they adjust to the new job? How quickly did they make an impact? How different was the new job to the previous job? This can give you a sense of how well someone will adjust to your environment. It can also be something to consider when you are thinking of moving someone within your organization to a new organization. Or filling a new position. Do you need to look outside? Or is there someone inside that loves a new challenge -- and most importantly, succeeds at it.

Motivation

I recently received one of those lovely (and I mean that) emails that tells you you're special. Like most of them, however, it's only goal seems to be propagation. This one goes something like this:

"If you've received this, it is because someone cares for you...
So please, tell the people you love and care for, before it is too late.

If you do not send it, you will have, once again passed up the opportunity to do something nice and beautiful. If you're 'too busy' to take those few minutes..."

Though it purports to praise, the receiver gets the message that they don't do nice things because they are too busy.

Now, what if, instead, it said this:

"So please, tell the people you love and care for, before it is too late.
The more people that you send this to, the better you'll be at reaching out to those you care about. "


How does this relate to quality? Because quality software is not just about the end product, but the motivation to create quality software. Many of these emails touch us because we need to know that people appreciate us. In the case of software development, those with the responsibility of making the schedules and testing the software need to be aware of the impact of 'how' they communicate to those who create the software.

Next time a project manager is determining the schedule for the next release, avoid phrases like 'never hits the date anyway' and 'It is just a minor addition'. (I have a whole list of forbidden words I'll post soon.) Next time a tester needs to report a bug, listen for the developer's response and tailor the next test and report for how the developer receives your communication.

I have seen teams with a reputation of chronically late delivery and bug laden code turned around because of good communication.

Just remember to avoid compliments that have threats and putdowns imbedded in them - undoing the good that they are intended to accomplish. These insidious messages wear us down. Who knows? Maybe they account for the industry statistic of computer professionals changing jobs every 3 years. Doesn't that lead to a brain drain, denying us the quality of someone very knowledgeable staying with the product?

Symbiosis - Making the Clever Beautiful

In one of my recent posts, I may have left the impression that OpenSource doesn't greatly enhance our software experience. But that's just not true. OpenSource is a great place to innovate and discover how to fulfill the needs of the software user. The initial OpenSource software is normally quite technical and clever, and one of the reasons I started this blog is that software pervades our life, and the true benefits of innovation are realized as it enters mainstream use. For that, the clever must be made beautiful.

So why did I title this symbiosis? I read a great article in Make magazine called "Make Like Picasso". It says that Picasso said that when you first make complicated things, they are ugly, and someone follows who makes them pretty. The article goes on: "functionally clever things deserve to be pretty". And I agree. I believe that we need symbiosis to achieve that.

Symbiosis is more than a complementary relationship. Two are dependent on one another. Without both, each fails.

Some of our best innovations have failed when this is not understood. In order to be successful, General Motor's original electric car, the EV1, needed California to pass a law that they were necessary. This happened. Further, the car needed to be useful to the public. This happened. Then GM needed to continue to find value in the car, expand it to the rest of the states and other markets. This didn't happen.

The Greeks believed that you could determine beauty through the Golden Ratio. With the Golden Ratio you can measure harmony. Harmony requires balance.

After the initial roll out of the EV1, GM did not have the same level of commitment to its success as the users. Balance did not exist. Without it, symbiosis did not exist. Without that, the EV1 failed. What that meant was that instead of GM being ahead of everyone as this gas crisis has hit, we have had limited choices in auto alternatives. Neither of us has been as successful as we could have been.

In contrast, as Open Source, and the Linux operating system especially, became more popular, companies like Red Hat understood that a partnership was needed to allow the mainstream to achieve the full benefits of Open Source. Software development is benefiting greatly from the symbiosis relationships developing between Open Source innovation and its application.

Finding design flaws

When it comes to quality software, sometimes it isn't immediately apparent that the quality has a flaw. It seems to work. It does what we need it to do. In fact, we're pretty happy with our software. Right up until the point that it has become integral to our lives, and it stops working. In this case, I"m talking about the underlying systems.

Security software, print drivers, disk drivers, document readers. Most of these pieces of software are written not to be noticed. And they do their jobs pretty well. And for them, we only notice them when they fail. And when they do, it can have a big impact.

In this entry, you'll find 7 principles for addressing design flaws for better quality.

The bridge collapse in Minneapolis was the result of a design flaw, they said. Budgets weren't put in place to address critical flaws once they had been discovered. The infrastructure of any country needs to be reliable, and unnoticed. And that costs money.

As software impacts more and more of our lives, the same level of rigor that used to go into evaluating our infrastructure needs to go into our software.

Borrowing from the discipline of building a bridge, infrastructure software design must be measured against standards and evaluated by independent experts who sign off on the design. Now it is reasonable to consider cost impact when evaluating designs. But as this article suggests, cost impact is weighed within the context of the design goals.

Where I live, the investigation into Boston Big Dig's tunnel failure cited: poor design specifications, paying contractors to test improperly installed anchor bolts and lack of consultation with tunnel designers before allowing contractors to drill through steel reinforcements in the tunnel roof.

What it identifies is three places where you can find design flaws:

1. review design specifications
2. use the proper test procedures
3. consult with original designers before making design changes

To these can be added lessons from the bridge collapse:

4. review old designs based on new information
5. identify time criticality for corrective action
6. budget for corrective action
7. publish findings

Expectations

A great deal of quality can relate to expectations. For example, I recently visited a new "New York Style Deli". Now, for anyone who has been to New York City, the deli could be expected to have fresh bagels, rolls and breads, a wide variety of cold cuts and hot meats, peppers, etc. all able to be put together quickly. Often, you can see what you are going to be getting. Apparently, the people who named this place had never been to a New York Deli. We found a display case with some bagels in plastic bags, obviously purchased from somewhere, the cooks not in sight, and a menu that didn't have all the items or prices. Hmmm.

As you might expect, the prices and food weren't that great either. When a customer makes a decision to buy something, it is based on what they might reasonably expect to get. In software, we talk about 'managing expectations', which normally translates into schedules, but I'm talking about actual quality expectations. What do I want, and will I get what I want? So, let's eliminate 'when' for a second.

And let's not even go to the fine print type of thing ("no setup required." "works out of the box.")Because there is meaning that can convey not just a feature, but an entire experience. What do you think of when you get a product from Apple Computers, for example. Ok, in that case, you do expect the "no setup required".

Here's another example. What do you think when you see this ad with the bathtubs outside? I'll tell you what I think. How can they possibly accomplish what the drug maker says is the purpose of the drug? Don't have your customers thinking that.

They can't be thinking "this isn't what it purports to be", nor can they be thinking "but I can't do what I want to do if it works this way." One of the biggest challenges for quality software is knowing what you really want to make and whether you can indeed make it work. And that is more than just a list of requirements. It requires seeing the picture and keeping it in mind while developing the product.

Unnecessary Features

We seem to have a problem in our society with extras. Fast food chains offer you a 'meal' and in a particular size. The sizes change. Cars on the lot come with 'extras', but they can't seem to find the basic car at anyone's lot. Software improvements contain items we just don't need.

We often hear that in order to be successful in the workplace, we need to accept that change is a part of it. And so the 40 hour work week earned on the sweat and blood of unions is fast disappearing. But in software, anyway, some of that extra work and programming is for extra features that the customers never said they wanted and rarely use. Features are determined by a cost benefit analysis, rather than determining whether new features are required at all.

Far from making the product more appealing, customers often feel forced to buy the only version available. But sometimes, there is blowback. Microsoft's Vista's is a case in point. Sometimes, the customer doesn't buy. Or the customer complains. Or Open Source becomes popular.

In fact, I sometimes think that features should face a battle such as we're seeing in the Democratic primaries before they are allowed in to a piece of software. Now you may think that Open Source would be the answer to that, but Open Source is more like, if you want it, you can have it (think Wendy's burgers), rather than a collective vote.

So, what's the answer? Rather than a constant march for new features, perhaps a new version that removes bugs from the existing version -- only. Just make it better. Just make it higher quality. What do you think?

The last test

You've got a test plan. You've followed the test plan. You've run the test cases.

...again...
...and again...
and again.

Finally, you have the final release candidate. You run through your tests.

Now it is time to go into production. And you get?

A printed farm bill sent to the White House only to find 34 pages weren't printed.

How did this happen? Simple fatigue and forgetting to have a clear implementation test? No one had a check list to check it? Or those that did had printed it so many times that they were sure it was there? Or they were rushing? Doesn't matter.

You must check the actual version -- the actual, actual version, not the one in the directory from which the actual version will be copied. You must have an implementation test that knows what you have to check.

What is just as interesting is how members of Congress and the White House are responding to this gaffe. And that also relates to Quality Software. But that's for another post.

American Airlines and Perception Quality

Recently, American Airlines announced that it would charge for all checked bags. This is a good example of negative perception quality. Even though American Airlines is using shared risk communication to explain this new charge, the impact will be negative quality.

In addition to a potential increased cost, there is the message that the company is hurting, and must take away one of the features that you have come to expect. This is similar to the software that 'unbundles' features and ask customers to pay for those features, when they had initially been free.

In the same way that customers with buggy software often find workarounds, customers will find more and more ways to stuff their carry on luggage, thereby potentially irritating existing customers who would not ordinarily check bags. This will lead to increased stress on the workers, who will already be more concerned for their jobs, in the same way that the focus on software companies ongoing cost-cutting efforts impact developers and quality staff. And staff focused on the security of their jobs are more likely to make mistakes and be less focused on quality for their customers.

Perception quality applies not only to customers, but to staff as well. It impacts the quality of what is developed, and the value of what is produced. And negative perception quality can have long term impact even if there is a remedy or reversal relatively soon after the blunder.

The software contract

I want to discuss the software contract - not the one we all ignore in commercial software. There are many software contracts for custom software. These are often the result of RFP (request for proposal), response to RFP, negotiations, all resulting in a contract.

What does your contract contain? There are key elements that should be found in the contract. I will list those that relate to the quality of the software and in a separate post (or series of posts) discuss how.

1. Time or dollar constraint

2. Testing requirement (this is broken further down by who, when, how much, linkage to acceptance and payment)

3. Software requirements

4. Relationship to the RFP

5. Modification responsibilities

6. Guarantee of quality (including definition of what that means)

7. Personnel assignments (including both customer and producer)

8. Signoff authority (including both customer and producer)

Gartner has an article on negotiating contracts here.

What is technical depth anyway?

I recently was talking with a company about technical depth. And it got me to thinking about resumes, interviews and general understanding about expertise.

The quality of any software is directly related to the abilities of the people involved in producing that software. So, when you're choosing your staff, you must find a way to understand if they belong on the team. Johanna Rothman addresses this in a series of blog entries called "Hiring Technical People."

Also, here is a rather lengthy discussion about technical skills for an IT Architect. What I like about it is how they use the term (do a quick search to find it in the article).

Technical depth is verifiable deep knowledge on a specific subject. A person should claim or not, technical depth on a particular subject. The specific subject should be required for the job, and noted as such.

What I would like to see remedied is a vague approach to an undefined 'technical depth' that wastes people's time and potentially impacts the quality of software. It takes two forms.

1. It is common to have 5 or 6 interviewers all with technical agendas which may or may not be clear to the interviewee.
2. An interviewee may imply technical depth on a resume that an interviewer may pursue to determine interviewee integrity, but which they may or may not be able to verify and which may or may not be relevant to the job.

Again, I would define Technical Depth as verifiable deep knowledge on a specific subject. A person should claim - or not - technical depth on a particular subject. A company should request technical depth on a specific subject required for the job.

Demos, Tests and Databases

This might be too much for one entry, but here goes. The art of testing usually involves some form of data and some configuration of machines, operating systems and other items that constitute the 'system'. Over the years, one of the biggest challenges is to recreate that actual environment as the customer would experience it. And it is a thorn in the side of most test professionals. How does it relate to quality software?

It relates to quality in terms of quality are a measure of customer perception, in terms of real dollars spent and in terms of the relationship between testers and developers that is so important to efficient scheduling.

1. Hardware costs money, so the same hardware used to test the software is sometimes shared with the people doing demos. Usually, but not always these people are marketing people. Consider when it is for beta testing using customer data and multiple customers see it during demos.
2. It is common for the data to become hopelessly out of date in the test area. So that even though you have the hardware you need, you don't have the real data. So, one of two things happens. You report bugs, which quickly get bounced back by the developers, because you were using bad data. Or, you don't find any performance issues, because your data set doesn't accurately reflect the size of the data set.

You may have heard that live data has been used to test passports - particularly presidential candidates information. This is actually quite common activity. Many times I will be told that live data, live systems (with test data) are used, especially for performance testing. What is it doing to the performance of the actual system?

Or, what happens during a demo when competitors get to see your customers data? Or, what happens to the test schedule when you can't get the system you need? Or it got hopelessly reconfigured -- again, with invalid bug problems.

There are solutions to this -- never share systems between departments, set up rigorous procedures for sharing systems, create transparency with your customers if you are going to use their data -- get their permission and allow them to audit your use of it.

What does quality mean to you?

This question has been asked in many times and many ways. Robert Pirsig famously wrote Zen and the Art of Motorcycle Maintenance and followed it up with another book called Lila whose main theme was the struggle of defining quality.

In software, the definition of quality is similarly elusive. I prefer the definition that says that you need to understand the core purpose of the software. And that can't break. This entry is about testing. There has been a lot written lately about how you can't test everything in today's software. There are lots of reasons for that. But that is for another discussion. Here, I want to say that all test planning needs to start from the core purpose. If you have software that controls a car wash, then first and foremost, the car has to be clean at the end. It can give you bad paperwork, and the customer may tolerate it, but the car must be washed.

In selecting tools, the size of the test team, and planning the rounds of testing, core is vital. I have designed some automated test suites, and each time, the first thing I wanted to write was core tests. So that, no matter how many changes happened to that software, core would still work. Because I believe that is what the customer needs and expects.

Project management is tough

Project management is a dirty job. It is all responsibility and tracking and taking the blame without the creative input and true control required for success. OK, maybe I'm being a little hard here. But too often I see project managers that get hit from all sides without the real authority to control what is going on with the project. Why is that? I wonder if you asked 10 people, would they come up with the same description of what a project manager is and does? More importantly, how do they interact with a project manager.

There's the Project Management Institute, with its standards, and that's great. On the day to day efforts, though, you have layered different methodologies, ideologies and disciplines in different departments that all come together for a single project. Can one person manage all that? In fact, how does one define a project? Who defines a project? And the fatal, "who owns this project?" For a project manager to be successful, the project manager must be entrusted with true authority. In return, the project manager must commit to a professional level of knowledge of what it takes to run a good project. That will bring job satisfaction to the hardest hit project manager.

About buffalo chicken cream cheese

There's a wonderful cafe that I go to after my morning walk. In addition to a large variety of coffees, they have healthy snacks. Recently, a couple have surfaced that triggered the Quality gene in me.

Now, what would you think if you saw buffalo chicken flavored cream cheese? I'll tell you what I thought -- the whole point of the cream cheese is to cut the bite of the spicy chicken flavor. If you mix the two together you defeated the purpose.

Not to be outdone, the healthy cookies I bought market themselves as reminding you of the taste of lemon squares or lemon meringue pie. But they make the same mistake. The flavor that is so nice is the balance between the meringue, the crust and the lemon all sitting separately in your mouth. If you mix them all together in the cookie, the flavor is changed.

Now, I know this is a blog about Quality Software, so how is food relevant? The lesson in my morning coffee break is to remember the purpose of the components. When you are developing requirements for a new product, consider how the items balance each other. Relational databases and object oriented development, for example, were meant to decouple components that were hard to reach without going through a number of steps to get there. But human nature being what it is, we are still seeing products where you have to do any number of things before you get to what you want. Or they put too many fields on the screen, before I have all the information for each of those fields. The latter is particularly true of defect tracking systems and check ins for source code control software.

I'd like my cream cheese after my buffalo chicken, thank you.

Waterfall and Agile

Does anyone remember when the test of a good software engineer/programmer/developer was if they vowed to never, never use 'goto'? Or insisted that they always clean up their memory allocations? But when you took a look at their code, their techniques for avoiding 'goto's or cleaning up memory made for some high maintenance spaghetti code.

Well, in the software development methodology, Waterfall seems to have befallen a similar fate. And the current favorite for a replacement is Agile.

I was thinking about this as I was upgrading some home software from the version developed in 2005 to the 2008 version. The reason I was thinking about this is that there is some software that businesses large and small only upgrade about that often, because they are core systems. And Waterfall was designed to control the development of vital systems, and Agile expects to roll out new versions of code in months.

I propose that when companies implement Agile, it should be done in conjunction with some good measurements. Determine the percentage of customers who implement the various versions that are released, as well as the number of versions that are released, as well as the overhead that those releases cost. Agile was not introduced to just hurry out as many versions as possible, which I think is sometimes happening.

In the same way that the occasional 'goto' produced more elegant code, some level of control, as came from Waterfall, is still needed for larger core products. In his post PRINCE2 + AGILE = Common sense?, Craig Cockburn makes a good proposal for combining the controls needed for a large project with the flexibility of Agile.

What ever happened to the rollback plan?

Over Christmas, I found myself saying "That's why I got into the Quality software business!" When you ship packages through USPS or UPS or FedEx, you get a tracking number that purports to tell you where your package is, every step of the way. One of them, consistently said "we will be receiving it from its origin on xxxx". Even after it was delivered! Now this worked great last year.

I happen to know that it is pretty normal to upgrade software once a year. Here's the thing - last year, the software worked great. What changed? I'm guessing a new version happened. And here is where it goes wrong.

When did we stop having a good rollback plan? When you have software that works, and works well, you should be very careful in your upgrade plans. And always, always be willing to rollback to the previous version that worked so well. And anyone writing new software needs to be ready to have that software rolled back. But too often, upgrades happen, and happen poorly. It could be personal, as when you upgrade your personal software, it changes the storage of the underlying data, and when there is a problem, you can't figure out how to go back. Explanations and documentation of what is being changed may, on the producers side be poorly written, or, on the installer's side, be poorly read! One of these things seems to have happened with holiday shipping this year.

So, whatever happened to the rollback plan?