Most people have probably already heard about the less than successful launch of http://www.healthcare.gov, often colloquially referred to as the ‘Obamacare website’. Bloggers and news agencies quickly jumped on the bandwagon to point out every piece of available evidence that this project is deeply flawed in every single way. Fingers have been pointing and names have been called. And let’s not start talking about what ObamaCare’s political opponents have said.
Certainly, there are plenty of differing opinions out there about what went wrong: Some will say that this is a reinforcing evidence that large software projects with too many people involved is a management nightmare that almost without exception, ends in failure until it hits version 3.0. While others will tell you that this is simply an expected outcome whenever the government embarks on just about anything. A third group will point to specific technical flaws that have emerged and it’s a clear indication that both management and the software engineers involved are simply bad people making kindergarten level mistakes.
So, what has this got to do with load testing? Quite a lot actually. As the makers of a very popular cloud-based load testing tool, we’ve always been advocates of tools and methodologies that leads to good and verifiable software quality.
Admittedly, we specialized in performance testing, but in this case it goes beyond that. Our opinion on what went wrong and what should have been done takes a non-political stand, in fact it’s pretty much neutral on development methodology and we definitely won’t call names. Just like in politics, our opinion boils down simply – it’s all about priorities.
Take a minute to think about the phrases ‘software quality’ and ‘verifiable software quality’. That additional word in the latter phrase means a lot and it changes everything. I can safely bet that this is something 100% of all software project managers covet, I mean, who wouldn’t? Yet it’s safe to say that less than 50% of all software PM’s can confidently claim that their projects have achieved it.
And why is that? Well, we’ve discussed it before here, when we briefly commented on our State of Web Readiness study. To begin, software quality doesn’t automatically materialize out of thin air in a project just because you have hired the crème de la crème of developers (how would you even define that?), let alone even the most experienced developers.
You will have good software quality if you make it a priority; not with a mere ‘Should-have’ but a ‘top of the line, grade A must-have’ type of priority. Then, when you’ve decided that quality is top priority in your project (again, who wouldn’t?), adding the concept of verifiable software quality is another leap.
Going from the intention of developing good quality software to measuring it is a big but essential step. A lot of development organizations around the world have taken this step and I would be surprised if any of them regretted choosing it. Surely, it involves some form of upfront investment to do it correctly but once you’ve taken the initial steps, your project will benefit from the fruits of your labour.
I’m sure that what I’m saying here is not new to anyone involved in the healthcare.gov project. In a software project north of 500 million USD there’s bound to be many software quality assurance tools in place already. If I should guess, I’d say that the problem with the healthcare.gov project was a problem of test coverage. Evidently, some areas weren’t tested at all, while a large portion of the project hadn’t been tested in all aspects as it should.
What about performance testing? Well, it should be obvious that a system that needs to handle tens of thousands of concurrent users needs to be tested for performance in general and specifically to be tested under load; not just at the end but throughout all development cycles.
In the news we’ve read about an internal 100-user test that was done just one week before launch that managed to bring the site down. It is apparent that load testing the entire site hadn’t been carried out correctly, or worse, not at all.
To round up, I will offer two pieces of unsolicited advice to the entire team behind healthcare.gov:
Number one, don’t panic! Panic is probably what brought you into here in the first place.
Secondly, commit to verifiable software quality and use tools. Tools that measure how you’re doing, tools to verify you’re on the right track and to help you find unacceptable quality. And when you realize you need a really good load testing tool, give us a call.
By: Erik Torsner (@eriktorsner)
Load Impact Blog 