The National Aeronautics and Space Administration (NASA) has been a topic of interest since President Obama took office more than a year ago and almost immediately ordered an independent review of NASA’s manned space program.  That review, carried out by the Augustine Commission, had nothing but bad news for the Administration, reporting “The U.S. human spaceflight program appears to be on an unsustainable trajectory.”

More recently—and perhaps in response to the Augustine Commission’s pessimistic assessment—the Obama Administration announced a budgetary shift for NASA, giving the agency more funds for R&D, while taking away funds from the planned return to the moon—“killing the Constellation program” in the words of this article by the Washington Post.  The article explains—

The old NASA strategy set a destination first, and developed technology -- rockets and a crew capsule -- to get there. The new strategy calls for pouring billions into space technologies without defining the destination. The idea is to create technological flexibility so that astronauts could potentially visit a variety of locations in the inner solar system, including the moon, near-Earth asteroids and possibly Mars or the moons of Mars. The ‘Flexible Path’ strategy was favored by the advisory panel appointed last year by President Obama to review NASA's options.

This article at SpaceNews.com is rather more optimistic; it quotes a NASA executive as saying, “This is a great budget.  It’s a major increase to the science budget.  It’s an especially major increase to the Earth science budget.”

While NASA wrestled with its future vision and mapping a path to manned space missions, back on Earth the Government Accountability Office (GAO) was busy criticizing its unmanned satellite missions, telling Congress that “NASA frequently exceeded its own acquisition cost and schedule estimates, even when those estimates were relatively new. In fact, 9 out of 10 projects that have been in implementation for several years significantly exceeded their cost or schedule baseline estimates—all in the last 3 years.”

In a report released on February 1, 2010, GAO assessed 19 “large-scale” unmanned NASA projects, of which 15 had entered the “implementation” phase.  Of those 15, GAO concluded that nine of them “experienced significant cost and/or schedule growth from their project baselines.”  (Four projects were considered to be in the “formulation” phase, and NASA did not provide GAO with formal cost and schedule information on them.)  GAO reported—

Based on our analysis, development costs for projects in our review increased by an average of over 13 percent from their baseline cost estimates—including one project that increased by over 68 percent—and an average delay of almost 11 months to their launch dates. These averages were significantly higher when the four projects that just entered implementation are excluded.  Specifically, there are 10 projects of analytical interest because (1) they are in the implementation phase, and (2) their baselines are old enough to begin to track variances. Most of these 10 projects have experienced significant cost and/or schedule growth, often both. These projects had an average development cost growth of 18.7 percent—or almost $121.1 million—and schedule growth of over 15 months, and a total increase in development cost of over $1.2 billion. Over half of this total increase in development cost—or $706.6 million—occurred in the last year. These cost growth and schedule delays have all occurred within the last 3 years, and a number of these projects had experienced considerable cost growth before baselines were established in response to the 2005 statutory reporting requirement.

See GAO’s entire report here.  The GAO report also noted that roughly $1 billion in ARRA funds was given to NASA, much of which was used to offset some of the cost growth it reported.

In its report, GAO reviewed six factors that it believed contributed to cost and schedule growth.  Those six factors were:

• Technology maturity (including maturity of critical technology as well as maturity of heritage technology)
• Design stability
• Contractor performance
• Development partner performance
• Funding issues
• Launch manifest issues

The report discussed each of the above factors as they apply to NASA’s projects.  More interestingly, the report devotes two pages of detail to each individual project.  Following are some “highlights” from the detailed project reports.

• Aquarius Project – Only 16 percent of engineering drawings released at Critical Design Review (CDR) milestone.  No engineering test models because of funding constraints.  Ten month schedule slip and $10.7 million cost growth.
• Ares I Crew Launch Vehicle (CLV) – “The Constellation program’s poorly phased funding plan has diminished the Ares I project’s ability to deal with technical challenges.”
• Glory Project – Only 69 percent of drawing released at CDR.  14 percent cost increase and 10 month schedule slip since rebaselining in FY 2009.  Currently, recent launch failure of Taurus XL rocket threatening current scheduled launch date.
• James Webb Space Telescope (JWST) – “The JWST project was re-planned in fiscal year 2006 after a $1 billion cost increase and a 2-year schedule delay on the project. In fiscal year 2009, the project established its baseline with a life cycle cost of $4.96 billion and a June 2014 launch date. This represents about a $500 million increase over NASA’s 2006 replan figures and has resulted in another 1-year delay of the launch readiness date.”
• Kepler – “Since being baselined in fiscal year 2007, Kepler’s development costs have increased by about 25 percent and its schedule has increased by 9 months. Project officials attribute the cost and schedule growth to many things, including a $35 million budget reduction in fiscal year 2005. This funding instability contributed to an overall 20-month delay in the project’s schedule and about $169 million in cost growth.”
• Mars Science Laboratory (MSL) – “MSL’s cost has grown over $660 million because of technological and engineering problems. This includes more than a 68 percent increase in development costs. The project is using a 25-month schedule delay to work on overcoming technical challenges with the actuators and avionics that were the primary drivers for the slip.”
• Solar Dynamics Observatory (SDO – “SDO’s design was not stable at the critical design review (CDR). Following this review, the project experienced nearly a 1,200 percent overall increase in the number of releasable drawings expected. Project officials said only drawings for in-house structures, such as propulsion systems, electronics, instrument ports, the high-gain antenna system, and the spacecraft, were considered at CDR. Drawings for the instruments were not included and flight drawings were only in draft form at CDR. The project estimated it released less than 63 percent of engineering drawings at each of their instrument-level CDRs.”

From one point of view, the GAO report is a litany of project management failures.  But from another point of view, the report tells various stories of individual project teams doing the best they could in difficult circumstances, trying to create state-of-the-art scientific instruments and manage complex systems, while being held to unrealistic budgets and almost impossible launch schedules.

Very few people care about the difficulties and the challenges of the NASA missions; they prefer to criticize the easy targets of cost growth and schedule slip.  Even though we here at Apogee Consulting, Inc. firmly maintain that projects can be managed to successful cost, schedule, quality and performance objectives, we also understand the difficulties faced by these project teams.  We are reminded of this saying by Theodore Roosevelt, spoken before the Sorbonne in 1910—

It is not the critic who counts: not the man who points out how the strong man stumbles or where the doer of deeds could have done better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood, who strives valiantly, who errs and comes up short again and again, because there is no effort without error or shortcoming, but who knows the great enthusiasms, the great devotions, who spends himself for a worthy cause; who, at the best, knows, in the end, the triumph of high achievement, and who, at the worst, if he fails, at least he fails while daring greatly, so that his place shall never be with those cold and timid souls who knew neither victory nor defeat.

On January 20, 2010, GAO offered testimony before the House Committee on Appropriations, Subcommittee on Defense (John Murtha, D-PA, Chair).  The testimony, which can be found here, had the rather misleading title of “Managing Risk to Achieve Better Outcomes.”   The testimony was not really about risk management; instead, it was more of a recap of various DOD acquisition issues that the GAO has been criticizing for years—most of which are rather obvious.

Highlights of the GAO testimony—

• The first opportunity to reduce acquisition risk is in the early planning phases.
• Risk can be reduced through selecting the appropriate contract type.
• Risks that are not effectively managed in the pre-execution phase of a contract may become costly problems during execution.
• Acquisition risk management is dependent on a “capable” acquisition workforce; yet “DOD lacks key information about the current number and skill sets of its acquisition workforce and what it needs.”
• Because DOD relies so heavily on contractors to support its acquisition process, and lacks insight into the roles its contractors are fulfilling, the Department cannot make effective decisions regarding its acquisition workforce staffing—thus “increasing the risk of transferring government responsibilities to contractors.”

The testimony reiterated familiar themes, such as—

Risk is inherent in any major acquisition, whether a weapon system or complex service acquisition. But it is only through the thoughtful management of risks throughout all phases of the acquisition process that successful outcomes can be achieved. Clearly, however, DOD has not adequately managed such risks. For example, its major weapon systems continue to take longer to develop, cost more, and deliver fewer quantities and capabilities than originally planned. … Despite decades of reform efforts, these outcomes and their underlying causes have proven resistant to change and, in fact, both DOD weapon system acquisition and DOD contract management have been on our high-risk list for nearly 20 years.

Much more interesting than the reiteration of simplistic observations is the summary of GAO bid protest statistics.  Although released separately, the testimony included an overview of the recent history of GAO’s “quasi-judicial” function as arbiter of bid protests.  (Readers will note that the U.S. Court of Federal Claims also performs in a similar capacity.)  We previously have wondered out loud whether the Department of Defense has “lost its ability to evaluate proposals fairly and in accordance with the written evaluation criteria.”  The GAO testified that—

There has been a substantial increase in spending on acquisition programs and services, while the number of civilian and military personnel in DOD’s acquisition workforce has remained relatively constant. Program offices have reported that workforce shortfalls have resulted in degradation in oversight, delays in certain management and contracting activities, and increased workloads for existing staff. To supplement its in-house acquisition workforce, DOD relies heavily on contractor personnel. For example, we have found that in some program offices contractor personnel outnumber DOD personnel.

Have DOD’s acquisition workforce problems contributed to degradation in the ability to conduct fair and equitable evaluations of bidders for major defense acquisition programs?  One way to assess the situation would be to look at GAO’s bid protest decisions, to see whether the sustention rate is elevated when compared to historical norms.

According to the GAO 2009 bid protests are up 20% over the prior year—1,764 vs. 1,458.  Comparing 2009 to 2005, bid protests are up nearly 40%--1,764 vs. 1,262.  (Readers should take GAO’s numbers with a grain of salt, however.  GAO counts every “docket number” as an individual protest; a single protest can have multiple docket numbers.  Moreover, the quantity of protests filed really tells us nothing.  Government spending also increased during that period, so one might logically expect an increase in bid protest activity.  In addition, GAO recently received expanded bid protest jurisdiction over task orders, which accounting for 49 of its 2009 bid protests) 

Sustention of bid protests is measured against “merit decision” (not the number of cases filed) and the numbers show that the sustention rate actually dropped during that period—from 71/306 (23%) to 57/315 (18%).  So rather than a degradation, the evidence points to an improvement in the Government’s ability to carry out its acquisition function!

Looking just at the Defense Department, the number of protests filed was up nearly 50 percent—from 706 to 1,050.  But as before, the percentage of sustained protests actually decreased from 18% to 12%.

GAO also reports an “effectiveness rate”—which is defined as a situation when a protester receives some form of relief from the Government agency conducting the procurement, even if it did not have its protest sustained.  Looking at the reported effectiveness rates, there has actually been some increase, from 27 percent (2005) to 45 percent (2009).  It’s difficult to determine what this means without further details.

So the statistics may be meaningful, or not.  We’ll leave any conclusion(s) to our readers’ judgment.  But we maintain that anecdotal evidence is mounting that DOD is having trouble evaluating bidders for major defense acquisition programs without opening the door to a successful bid protest.  As GAO notes in its report, such recent programs as CSAR/X, Aerial Refueling Tanker, FMTV, and TRICARE were interrupted by successful protests by the losing bidders.  And as we have noted with respect to two of those programs listed by GAO, the errors cited by GAO were of the “blocking and tackling” variety, and not of the “difficult and complex” variety.  We hope DOD gets its act together, soon.