Recently we inquired into the program management problems at the Department of Homeland Security.  In the past, we asked similar questions of NASA and DOD.  Now we turn our attention to the Missile Defense Agency (MDA), asking why “DOD’s largest single acquisition program” is having performance problems.  Related to that overarching inquiry are two other questions—(1) why do seven of 14 MDA prime contractors have noncompliant EVM systems, and (2) why do two of 14 MDA programs assessed have such unreliable EVM data that GAO was unable to “identify significant performance drivers or forecast future cost and schedule performance.”

We shall start our investigation with this GAO report, entitled “Missile Defense Program Instability Affects Reliability of Earned Value Management Data.”  In that report, GAO made its annual assessment of progress made by the MDA in developing and fielding the nation’s Ballistic Missile Defense System (BMDS).  The report supplements a previous GAO assessment.

By way of background, the GAO report states—

MDA’s mission is to develop an integrated and layered BMDS to defend the United States, its deployed forces, allies, and friends. In order to meet this mission, MDA is developing a highly complex system of systems—land-, sea- and space-based sensors, interceptors and battle management. Since its initiation in 2002, MDA has been given a significant amount of flexibility in executing the development and fielding of the BMDS.

Unlike other major defense acquisition programs, the BMDS has been exempted from the requirements of 10 U.S.C. § 2435 (which requires the programs to establish the total scope of work and total cost baselines), because the Secretary of Defense “delayed” entry of BMDS into DOD’s formal acquisition cycle.  The lack of an established baseline prevents GAO from evaluating progress of the program as a whole; instead, it has to look at each BMDS program on an individual basis.  In this report, GAO looked at 14 MDA programs.

The first thing GAO noticed was that two important programs—Ground-Based Missile Defense (GMD and Targets/Countermeasures—couldn’t be assessed.  The two programs couldn’t be assessed because their Earned Value Management (EVM) data “were not sufficiently reliable to analyze” the contracts’ cost and schedule performance.  What’s going on here?

First, the two programs each had baselines that were “no longer representative of the program of record.”  With respect to the GMD program, GAO reported that the contractor (Boeing) had “experienced difficulty” incorporating “numerous changes to the program and [resulting] modifications to the contract.”  GAO reported—

For example, although the GMD program experienced a $1.3 billion dollar restructure in 2007, another major restructure beginning in fiscal year 2008 for over $500 million that was completed in fiscal year 2009, and a third in fiscal year 2010 for over $380 million, the GMD program has not conducted an IBR [integrated baseline review] since December 2006.  DOD’s acquisition policy states that an IBR is to be conducted within 6 months after contract award, exercise of contract options, or major modifications to a contract.  DCMA officials told us that the GMD program had an IBR underway following the restructure that began in fiscal year 2008 and completed in fiscal year 2009, but in May 2009 the program was again redirected and the baseline review was cancelled.  The Director, MDA explained that some of the GMD program’s baseline instability from frequent restructures was related to the changing GMD role in European defense. … The Director told us that these European capability requirements changes drastically affected the GMD program as a significant amount of work had to be restructured.

With respect to the Targets and Countermeasures program, the prime contractor (Lockheed Martin) was “unable to update its baseline because of numerous program changes.”  GAO reported—

In September 2007, when the delivery order for the launch vehicle-2 was approximately 60 percent complete, Lockheed Martin signaled that its baseline was no longer valid by requesting a formal reprogramming of the effort to include an overrun in its baseline for this delivery order. MDA allowed the contractor to perform a schedule rebaseline and remove schedule variances – but did not provide any more budget for the recognized overrun in the performance measurement baseline. As a result, DCMA reported that the performance indicators for this delivery order, needed to estimate a contract cost at completion, were unrealistic. According to the Director, MDA did not believe the contractor had justified that there was a scope change warranting additional budget in the performance measurement baseline. He said he believed doing so would mask problems the contractor was experiencing planning and executing the contract which he identified as the issue as opposed to changes in the contract’s scope. According to the Director, one example of the issues the contractor was experiencing on this delivery order included a failure rate of 64 percent on production qualification components. … In addition … program changes since fiscal year 2008 on one delivery order included over 20 contract changes to the scope of work or corrective actions to quality issues.

GAO noted that seven of the 14 programs it reviewed were managed by contractors whose EVM systems had been assessed by DCMA as being “noncompliant” with the applicable criteria of the ANSI/EIA standard governing earned value management systems.  Despite this situation, GAO used the EVM data to evaluate the other 12 programs.  It noted that “We reviewed the basis for the noncompliance and unassessed ratings and determined that” the EVM data was reliable enough “for our purposes.”  This finding, of course, begs the question of why DCMA evaluators would find the EVM systems to be inadequate while GAO found the EVMS’ outputs to be good enough.  Some in industry have accused DCMA evaluators of being overly picky in their evaluations … but that topic is probably better left to another article.  We’ll leave it with an example of GAO’s comments—

For example, the EVM system of the STSS contractor Northrop Grumman was deemed noncompliant because of two low-level corrective action requests related to issues with other contracts that did not materially affect the performance baseline for the STSS contract we assessed. Also, the C2BMC’s contractor Lockheed Martin Information Systems & Global Services received a rating of noncompliant during 2009 because of a corrective action request that stated that major subcontractor efforts were not specifically identified, assigned, or tracked in the organizational breakdown structure. However, after the noncompliant rating was given, DCMA reversed its decision and decided to close the corrective action without requiring the contractor to change its methods.

Looking at the 12 programs GAO felt it could review, it noted mixed results.  Some programs performed adequately while others experienced cost growth and schedule slips stemming from such issues as “technical complexity,” “quality issues,” “unanticipated design changes,” “late receipt of hardware and production-level drawings,” etc.  For example, with respect to the Command and Control, Battle Management, and Communications (C2BMC) program, GAO reported—

These budgeted cost overruns are driven by increased technical complexity of Spiral 6.4 development, and more support needed than planned to address requests from the warfighter for software modifications. The $4.2 million of unaccomplished work on the agreement is driven by efforts in the Part 5 portion of the agreement, including delays in system level tests, late completion of C2BMC interface control document updates, and unexpected complexity of algorithm development and network design.

GAO also noted that the STSS program (managed by Northrop Grumman) was on schedule, but nearly $73 million overrun against budget.  There were various reasons attributed to the cost growth, including slippage of the launch dates.  GAO asserted that—

If the contractor continues to perform as it did through September 2009, our analysis projects that at completion in September 2010, the work under the contract could cost from $620.9 million to $1.6 billion more than the budgeted cost of $1.6 billion.

Similarly, the development contract for THAAD (Terminal High-Altitude Area Defense, managed by Lockheed Martin) has experienced cumulative cost overruns of $262 million and is also behind schedule.  GAO reported—

The contractor attributes overruns to the missile, launcher, and radar portions of the contract. The missile’s unfavorable cost variance is driven by unexpected costs in electrical subsystems, propulsion, and divert and attitude control systems. Also contributing are issues associated with the optical block, range safety, communications systems, and boost motors. The launcher has experienced cost growth because of inefficiencies that occurred during hardware design, integration difficulties, quality issues leading to delivered hardware nonconformances, and ongoing software costs being higher than planned because of rework of software to correct testing anomalies. These problems resulted in schedule delays and higher labor costs to correct the problems. In addition, cooling and power issues with the radar have contributed to overruns with the prime power unit. Numerous fan motor control system redesigns and retrofits for the cooling system drove costs by the supplier. Inexperience with building a prime power unit and a limited understanding of the true complexity and risks associated with the system led to significant cost growth and delivery delays.

As we have noted with respect to other Executive Agencies, MDA seems to have experienced the standard challenges associated with developing technically challenging weapon and defense systems.  Requirements changes drive design changes, which affect manufacturing schedules.  Managing changes requires staff and other administrative resources, and often requires technical personnel to take time away from their program “day jobs” to support the administrative change management processes—meaning the program work doesn’t get accomplished as planned.

This GAO report seems to confirm what we’ve been hearing from industry—that DCMA EVMS functional specialists are nit-picking and looking for reasons to withhold EVM system approvals.  GAO was able to find reliable data to evaluate, despite numerous so-called noncompliances.  We hope DCMA quits treating EVMS like Government Property control systems, and develops a sense of materiality and proportionality in its system reviews.

On the two MDA programs with unreliable program baselines, we look to MDA itself as the culprit.  On one program, numerous program restructurings appeared to have significantly impacted the prime contractor’s ability to maintain baseline control.  On the other program, it was MDA’s decision not to revise the baseline when appropriate to do so, that led to the situation where GAO felt the baseline no longer represented a meaningful point from which to measure performance variances.  In addition, GAO noted numerous program changes that impacted the contractor’s ability to maintain baseline control, of such impact that GAO apparently believed the baseline should have been revised.

Here’s the lesson in all this—at least from our point of view.  If you have a technically challenging, complex development program—especially one where requirements are fuzzy and likely to change over time—then you need to expect significant changes.  You need to plan for the changes, and have a plan to manage them.  You need to anticipate the administrative resources necessary to identify, process, and incorporate changes into both the baseline and the contract.  And you need to anticipate the technical resources involved in that process as well.

Change control is one of the key processes that distinguish contractors in the marketplace.  Contractors that have robust change control processes, and that proactively manage the inevitable changes, do better than those that do not.  Period.

So how is your change control process working out for you?