A reader writes:
Part of my job is to do performance testing and analysis (mostly analysis) for clients running large web applications, some of whom are government agencies. Most of the time, projects are appointed some generic “Project Manager” throughout the whole development lifecycle (from design to production). The project manager’s sole value is to push people and process through bottlenecks to deliver the application on time and under budget. The good project managers understand the complexities of the application and what new data points mean to the project dependencies. They will build dynamic deadlines and breathing room into the project plan.
The bad ones will tune out info and march on to the deadline despite mitigating information. They fear not being able to achieve on time delivery, and will therefore deliver at any cost, hoping that things just work. They see stubbornness and pushiness as “what it takes to get the job done”. They create a culture where devs are afraid to speak up about needing more time to do things right. This is likely the personality assigned to Healthcare.gov. A dead giveaway of this situation is in this article, which states testing was done for two weeks! Our company demands that the project plan includes minimum six weeks for performance testing and analysis (which includes script building, dummy data population, testing accounts, fully integrated production simulation, and code and traffic instrumentation). In addition, any bottlenecks identified identified during testing will have to be addressed by the devs, and tested again.
Is Sebelius culpable? I really don’t think so. This really is squarely in the hands of the CIO at CMS (and everyone downstream to the project manager).
This is pointing at a bigger problem that affects almost all large institutions, and the way they role out new apps. The CIO is suppose to understand the way this works.
1) You have to phase rollouts for something this big and complex. The comparison to Facebook, Amazon, and Google is apt with respect to scale, but not process. And for good reason. Those three companies do not push out major updates and features to everyone at once. You phase it in, precisely to work out the bugs.
2) To help with above, you have to get away from “waterfall” process approaches to SLDC (software development lifecycle). The newer frameworks allow for a tighter integration btwn development and operations to fix problems faster, test changes, and do more intelligent delivery rollouts (phased).
3) I guarantee this was built on old (early 2000s) technology (IBM websphere, MS IIS, Oracle, SoftwareAG WM Integration). These older products were never meant to scale to the size of this app. The newer platforms (Apache Openstack, etc) are much more easily scaled than the static platforms mentioned above. But the govt and legacy enterprises never seem to grasp this. Apps in “the cloud” are mostly run on newer software platforms. But IBM, CGI, and all the other big outsourcing groups only develop on these older platforms, mostly using mediocre dev/engineering talent that relies on crappy certs more than real engineering talent.
I just don’t see Sebelius being responsible for this screw up. There is a lot that DHHS does outside of this – NIH, FDA, Medicare, Medicaid. If Sebelius goes, she is far enough removed from the location of the actual problem, that her departure would mean absolutely nothing. And the actual problems would persist.
And please keep in mind, nobody is dying here. This is a web application having issues, that will eventually get resolved, as more data becomes visible.
So we have the impossibility of getting a successor confirmed and the fact that she is not directly responsible for the clusterfuck. So who is the CIO at CMS? Update from a reader:
Seriously? 30 seconds on the google: “Tony Trenkle is the Chief Information Officer (CIO) and the Director of the Office of Information Services (OIS) in the Centers for Medicare and Medicaid Services (CMS)”
The Dish 