Sunday, November 23, 2008

Traditional Quality Assurance is Wasteful - Building Lean Teams

The traditional view of a software development team almost always includes some form of a quality assurance team. The classic QA team does some sort of formulated inspection of the product, balancing risk, time and resources with an end goal of eliminating as many defect leaks as possible.

The huge myth with this traditional view, is that the quality assurance team is your primary defense against delivering defects to your customer. In actuality it's far from being the most effective prevention. Building quality in to the product is the single best defense against introducing defects into a live system. Teams with agile approaches including unit tests and continuous integration are by nature building quality in.

There's a huge amount of waste in the traditional quality assurance team. So much time is spent making sure they are covering all the bases. You need to make sure that everyone on the team knows the product well enough to write detailed test cases. Even once a decent functional understanding is developed, there's still so much technical knowledge missing. What code is shared? Where could some concurrency issues occur? Those kinds of questions are rarely something that is answered by this kind of team.

It's also extremely difficult to staff a traditional quality assurance team. There's extremely high turnover in the position and general poor quality pool of candidates. To combat this, usually this function is shipped offshore, exacerbating the inefficiencies.

So many other inefficiencies stem from the formation of a QA team. For instance, how can a QA team write test cases if they don't know what the software is supposed to do? This leads to big up front designs (BUFD) and puts even more pressure on the design/development gap. Another inefficiency is the amount of effort that goes into maintaining the relationship between developers and QA. There's always a tension between the two teams that usually stems from under-communicating and a lack of mutual understanding.

If you've got a development team that's good about writing unit tests, uses continuous integration, and has a good grasp on unit test coverage, then you'll almost always find that a QA team offers little or no help in the quality of your application, and is far more of a distraction from getting things out the door. You may feel like arguing against the quality assurance team is an argument against quality, but it isn't. It's an argument for a lean team that builds quality into software that's getting out the door.


Anonymous said...

I get your drift, but what's the answer? Unit tests and continous integration builds don't find all of the issues, alone. What would be the make-up of your ideal team? Would there be no business analysts or dedicated QA people?

If not, how do you address the assumptions and biases that the person who built something has when testing it? (oh, I know that part works...)

Lots of anecdotal evidence says that when developers are left to test something, without a 3rd party QA, there are many more issues with the system, and you end up a frustrated users. That result can simply be a result of the ownership structure and exceptations on the team, though. If developers "own" the product from requirements to design to development to testing and support, the quality of self-testing may improve - but how much? And the success of this type of role depends heavily on the capabilities and desires of the individual filling it.

Jim Fiorato said...

Yes, in my opinion, the make up of the ideal software development team consists of a very technical leader and software developers. Strong ones. Very strong ones.

Regarding the "I built it, it works fine" complex, I think that people make that out to be far more pervasive than it actually is. In the cases where you've got the creative/productive guy, who builds great stuff, but suffers quality issues, the leader has to call them out on those issues, and make them better. In my opinion this is a far better trade off than non-value add testing with poor knowledge of the product.

I think there's a level of accountability that gets lost when this traditional concept of QA is in place as well. People tend to think "QA will look at it", and the level of thorough development testing decreases. I think this is one of the factors in why the anecdotes that you mention support this traditional view.

You're right, capability is everything. This is the reason you're trying to hire the best developers out there. You want creative thinking, attention to quality, and discipline in a lean environment. You've got all you need right there.

Anonymous said...

I've been in a couple environments along the lines of what you're describing (developers are effectively responsible) where quality was kept very high and customers (generally - of course, not always) were not finding our bugs. I think it works for the reasons you've laid out: There were more automated tests, additional (manual) QA was done by the devs + (your whole point) the teams were small and really high quality.

That said, the system that I've seen work best was as we've just described above + there was QA done by actual users. In this case the "actual users" were people working on non-dev teams in the company who needed intimate/everday knowledge of the app to do their jobs. AND it was done frequently and at least semi-formally.

I really feel that the best solutions are going to involve some combination of automated testing + manual testing by devs + qa of some sort (by non-dev team). That's not to say it's 33/33/33 though... I still haven't seen it work with a team that actually had "QA" in it's title.

Tom Wessels said...

I completely agree with you that quality has to be built into the product. The entire company must value defect prevention over defect identification. However, I strongly feel that QA is an essential part of a software development team.

Testers approach the product in an entirely different mindset that us developers cannot fully appreciate. Developers have a creative/constructive mentality, whereas testers have a destructive mentality. We're constantly looking for solutions while they're constantly looking for problems. These diametric viewpoints complement each other perfectly.

The problem is fully utilizing these different viewpoints. Again, a company first has to value defect prevention over defect identification, which ensures that everyone feels responsible and personally invested. But the feedback loop also has to be shortened. There will always be a disconnect if QA has to wait for developers to finish all features before testing commences. The back-and-forth nature of short iterations increases communication and product understanding between developers and testers.

That's the one aspect of the bug-fixing phase that I like. During bug-fixing, I'm constantly communicating with BAs, testers, and other developers to clarify requirements, explain functionality, refine scope, etc.--resulting in a high level of synergy throughout the entire team. It's just unfortunate that we have to wait to achieve this synergy until a release is over halfway over.

Jim Fiorato said...

I think it's important to reiterate that what I'm talking about in this series posts is building lean teams of really strong developers. I'm NOT talking about building teams of commodity developers to churn out features. There's a place for traditional qa in some companies.

I want to see the mindset that QA has but strong developers don't, but I can't. Every developer needs to know how to thoroughly test their code. Maybe I'm generalizing what you said about the differences are between QA's mindset and Development's mindset, but I don't think it is possible for you, as a developer, to build quality in if you only think in terms of solutions and not problems.

What I want you to think about here, is an environment where you own both the design of the feature, and the quality of it. 100% your responsibility. This makes you as the developer intimate with the feature, able to take creative liberty with it, and feel emotionally responsible for it's success.

We spend months and months looking for qualities in developer candidates that are very smart, creative, and disciplined. Why in the world would we only let them write code from a blueprint?

Mark Fletcher said...

If you want to cut out a QA team completely why not have other developers on your team test your code after you've tested your own code?

You wouldnt conduct a code review of your own code, so why do the same for testing? In teams that practice code reviews, developers take great pains to make sure that their code is of good quality, because they know their peers are going to be looking at it.

Similarly if you know one of your peers is going to double check what you've just tested, you're going to make sure that theres no silly bugs for them to stumble upon.

I think this is a neat solution to what Toms points out namely tackling the developers mindset to testing. It's either "I'll kick over to QA to pass" or its "I'll test it just enough to pass a quick inspection".

Developers can be rather devious, and I've known several developers who can test what appears to be working code, and come up with bugs that make you say "WTF!!".

The only drawback is added time due to manual testing, but if its something that is re-occurring, it might just be worth automating.

Jim Fiorato said...

Peer QA is perfect. Two days before launch, get everyone checking out the app. Meet up before hand and think about the things that have changed and brainstorm on risks.

My point is that WTF moments always happen, even with a traditional QA team. Eliminate that waste and put the ownership of quality into the developer's hands. Let your team of smart engineers figure out the best way to test efficiently.