
We’ve
asked this question before, about other agencies and departments of the
Executive Branch of the U.S. Government. Sometimes we imagine asking
the question wryly, or archly, or sometimes with a cynical and knowing
smirk. Other times we imagine asking the question in a plaintive or even
despairing voice. We see Grandma Jane from Wichita or Bismarck, looking
at the sky in supplication. We see Joe Six-Pack, wondering why his
hard-earned taxes don’t get spent wisely.
(We are, perhaps, overly dramatic.)
In
other words, this is the latest installment in an on-going series that
(hopefully) probes the ability of the Federal government to manage its
contractors and execute its programs. A quick review of our News Archive
will lead you to other articles in the series, each starting with the
word “Why”—which you can imagine in any voice you like.
Today
is the Federal Bureau of Investigation’s turn in the barrel. What
brings this respected law enforcement agency into our cross-hairs? For
starters, try this article
at FederalTimes.com. It reports that the FBI’s “Sentinel” project is
“is $100 million over budget and nearly two years behind schedule,”
according to a report released by the Justice Department’s Office of
Inspector General (DOJ OIG), which has cognizance over the FBI.
Here’s a link to the DOJ OIG Audit Report.
As
FederalTimes.com reported, the Sentinel project “is intended to replace
the FBI's outdated Automated Case Management System. When fully
implemented, it will provide FBI agents and analysts with a web-based
case management system to manage evidence, automate document review and
approval processes, and use expanded search capabilities.”
The
FBI expected to implement Sentinel in four overlapping phases, each
lasting 12 to 16 months. Each phase was intended to provide a
stand-alone set of capabilities upon which subsequent phases would add
further capabilities. The project’s first two phases were budgeted at $306 million; the FBI spent $405 million.
The
entire project was scheduled to be completed by December 2009 at a
total cost of $425 million. The project was rebaselined, and the
at-completion estimate was increased by $26 million, based on project
status in October 2007, when the first phase was completed. Now the
project is scheduled to be completed in September 2011 for a total cost
of $451 million.
In
other words, the FBI has asserted (via its budget) that it can complete
the final two phases of the work for between $20 and $46 million. And,
in the word of the DOJ OIG, “we believe that the most challenging development work for Sentinel still remains.”
However, MITRE (who was hired by the FBI to independently assess the project) calculated that it will take the FBI another $351 million
to complete the project—for an at-completion cost of $756 million
(which would represent an overrun of 178% against the original project
baseline budget of $425 million). Oh, and MITRE said it will take the
FBI another six years to complete the project.
Naturally, the FBI doesn’t agree with MITRE’s assessment. According to the FederalTimes.com article—
‘We
believe that the interim report does not accurately reflect the FBI's
management of the Sentinel project, and fails to credit the FBI with
taking corrective action to keep it on budget,’ according to a FBI news
release. The agency noted that thousands of its employees are using the
system to draft interview reports, send leads and manage their
caseloads. … The FBI says Mitre estimate's assumes a ‘worst case
scenario for a plan that we are no longer using.’
One of the planned corrective actions (according to the DOJ OIG) is that “the
FBI will assume direct management of Sentinel development and
significantly reduce the role of Lockheed Martin in developing
Sentinel.” More on that innovative approach in a bit.
What
went wrong? As usual, there were a number of decisions and factors that
led the FBI (and its prime contractor, Lockheed Martin) to this point.
Let’s look at the DOJ OIG report for some details.
On
December 2, 2009, the FBI conditionally accepted delivery of Sentinel’s
Phase 2, Segment 4, which included three of the eight electronic forms
expected to be delivered in Segment 4, and their associated workflow.
The FBI conditionally accepted this segment despite knowing that what
was delivered had serious performance and usability issues and had
received overwhelmingly negative user feedback during testing with FBI
agents and analysts. As a result, the FBI did not deploy Segment 4 to
the FBI’s agents and analysts when it conditionally accepted it in
December 2009.
Then,
on March 3, 2010, the FBI issued a partial stop-work order to Lockheed
Martin for portions of Phase 3 and all of Phase 4, and also returned
Phase 2, Segment 4 to the development phase from the operations and
maintenance phase. FBI officials stated that the purpose of the partial
stop-work order for Phases 3 and 4 was to focus Lockheed Martin’s
efforts on delivering Phase 2, Segment 4 in a form that the FBI would
find acceptable.
On July 26, 2010, the FBI deployed Segment 4 to FBI agents and analysts.
We
can see from the foregoing that the project experienced an eight-month
delay because of user satisfaction and system usability issues. But what
we also see is a Government project management team that is indecisive
and lacks leadership. First, they “conditionally accepted” the
deliverables and three months later decided to “return” the accepted
items back to “the development phase” for redesign and rework. Another
five months passed before Lockheed Martin could rework the electronic
forms to the satisfaction of the users—which indicates (to us) that
there was some very extensive rework done. I.e., it was not a quick fix.
So why were these forms initially accepted? One can only wonder.
What else did the DOJ OIG report?
As
of August 1, 2010, the FBI had not decided on an approach for
completing Sentinel, and FBI officials did not provide the OIG with
detailed descriptions of the alternatives under consideration for
completing Sentinel. At that time, however, the FBI Chief Technology
Officer stated that the alternatives under consideration would allow the
FBI to complete Sentinel within its $451 million budget by re-using
portions of successful FBI IT projects, including Sentinel, taking
advantage of technological advances and industry best practices, and
increasing the reliance on FBI personnel to develop Sentinel. Yet, the
Chief Technology Officer acknowledged that his estimate did not include
the cost of maintaining Sentinel for 2 years after its completion –
costs which had been included in all previous Sentinel budgets.
Yeah,
about the FBI’s official at-completion estimate: We understand that one
way to come in on budget it to forget to account for certain costs in
the estimate; however, that’s not the way it’s supposed to be done.
Here are some other facts about the FBI’s project management to consider, as reported by the DOJ OIG.
-
[The
DOJ OIG] found that the FBI has either limited in scope or eliminated
several project management activities that were designed to help it
monitor the progress of Sentinel’s development. … For example, in
December 2009, the FBI discontinued Sentinel’s Project Health
Assessments. Performed by the FBI’s Enterprise Requirements and
Assessment Unit, these assessments provided an independent assessment of
Sentinel’s cost, schedule, and scope.
-
The
FBI stopped EVM reporting for Phase 2 in December 2009, and the EVM
reporting for Phase 3 has not complied with OMB guidance since May 2010.
EVM is an important risk management tool for major capital investments
that measures the performance of a project by producing cost estimates,
evaluating progress, and analyzing cost and schedule performance trends.
… The FBI said that EVM and Project Health Assessments were
discontinued as a result of the March 2010 partial stop work order
because there was no schedule or baseline against which the FBI could
measure its progress. While we [the DOJ OIG] agree there was not
baseline to measure against, according to DOJ policy on implementing
EVM, once the FBI realized that Sentinel was significantly behind
schedule and over budget, the FBI should have established a new baseline
for measuring Sentinel’s cost and schedule performance.
Going
forward, the FBI told the DOJ OIG that it had a plan to assume more
direct control of the project, so as to bring it to completion on-budget
and on-schedule. According to the DOJ OIG—
Overall,
the FBI plans to reduce the number of contract employees working on
Sentinel from approximately 220 to 40. The FBI said that, at the same
time, the number of FBI employees assigned to the project will also
decrease from 30 to 12. The FBI asserted that this new, agile approach
will streamline decision-making processes and allow the FBI to deliver
Sentinel within budget.
The DOJ OIG enumerated nine concerns it had with the FBI’s proposed approach. Moreover, it reported that—
In
September 2010 Carnegie Mellon’s Software Engineering Institute (SEI)
performed an independent review of the FBI’s new approach for completing
Sentinel. The
SEI stated that the FBI’s decision to attempt an agile approach is ‘a
positive step toward improvement over the prior development approach.’
However, SEI expressed similar concerns to ours, including the largely
undocumented details of the FBI’s new plan, the FBI’s inexperience in
using an agile development methodology, and the unknowns concerning the
viable state of and path forward for the technical design of Sentinel.
To conclude, the DOJ OIG said that it had “significant
concerns and questions about the ability of this new approach to
complete the Sentinel project within budget, in a timely fashion, and
with similar functionality as what the Sentinel project previously
sought to provide.” Other than that, they were fine with it.
From
our perspective, this is a cautionary tale about lack of project
management skills, lack of rigor in measuring project status and using
that information to make an accurate estimate-at-completion, and a
corrective action plan that smacks of an alternate reality. Other than
that, we’re fine with it.
On a more positive and less sarcastic note, this article is our 300th
blog post on this website. That’s about three novels worth of
over-the-top strident and politically incorrect hyperbole. Thanks for
your support.