The FBI Software Upgrade That Wasn't 381
Davemania writes "Washington Post reports that the FBI's attempt to modernize their department has once again failed. The 170 million dollar Virtual case File system, the agency's second attempt to go paperless is reported to be useless. The finger seems to be pointing at the FBI leadership, greedy contractors and bad software management." From the article: "It appeared to work beautifully. Until Azmi, now the FBI's technology chief, asked about the error rate. Software problem reports, or SPRs, numbered in the hundreds, Azmi recalled in an interview. The problems were multiplying as engineers continued to run tests. Scores of basic functions had yet to be analyzed. 'A month before delivery, you don't have SPRs,' Azmi said. 'You're making things pretty. . . . You're changing colors.'"
Government Inefficiancy (Score:4, Insightful)
Paranoia! (Score:5, Insightful)
Yes. I'm slightly paranoid.
Maybe it's just me, but... (Score:3, Insightful)
Project management in outer space (Score:5, Insightful)
'A month before delivery,' Professor Knuth said looking up through his spectacles 'you can start implementing it if your correctness proofs are complete.'"
Ha! Welcome to the real world, guys.
Re:Paranoia! (Score:4, Insightful)
Why not just hook it all up to a search engine? (Score:4, Insightful)
Non CS people who commission custom software development often have no clue how expensive their ego driven non-standard features can be.
Re:Oblig. 'Fight Club' quote: (Score:5, Insightful)
Re:Government Contract$ (Score:4, Insightful)
What world does this guy live in? (Score:5, Insightful)
Wow... I have never, ever seen a software product that wasn't working on QA bug reports right up to the minute the gold disc is burned. And afterwards, of course, working on all the pre-release bugs that had been classified as 'known issues'.
Re:Government Contract$ (Score:3, Insightful)
And this would make it very difficult to get companies to do government contracts in the future.
Perhaps they should have taken in past history as well as cheap price, when deciding on contractors.
Re:Government Contract$ (Score:3, Insightful)
You seem to have left out a step: The government agency changes the requirements after the bid is awarded, usually in the user interface. If you're a smart bidder, your low bid was only to cover the original specification... and any modifications are extra.
This might be a case of a contractor sucking at the government teat, but lets not forget that clueless PHBs and design by comittee can also run up the cost of a project without producing results.
Re:Government Inefficiancy (Score:5, Insightful)
Outsource, and this is what you get. They must hire MBA's. Really, sensitive government data projects like this one should never be outsourced, if only for national security reasons.
Re:Most IT Projects Fail (Score:3, Insightful)
I would like to discuss this in some detail. Congress Critters are you listening?
The issue here is simple and almost sinfully so. If you are to get a job working for the Government, it doesn't matter which agency, you have to provide your quailification credentials. We see them on looking for a job as a list of qualifications. This applies to contractors who supply the government as wee. This becomes a list that looks like a laundry list of the history of the agency. We programmer types will know these very well. The list goes like this:
Programmer with 15+ years Experience in Matlab.
Must have 15+ years in Military Logistics with US Army SMDC
and the list goes on. There are only two problems with this listing. Both of these cannot exist. The only person who can qualify with these "pseudo credentials" is somebody who has just retired from the army and frankly even then it is a fiction. The result is that old unqualifed failure that was just booted out of the ranks for incompetence is now the only person qualified for duty.
In the FBI this is worse not better. The results are that the FBI remains unable to respond because it cannot recruit new blood to infuse it. The situation in contracts is even worse since the bureaucrats in the Government reserve the right to pick and choose the people who will work with them on contract. The contractor doesn't get to pick his people! This makes absolutely sure that in the post 9/11/2001 world only those who failed us before 9/11/2001 can ever be "Qualified" to do any work. They are the "Experts" we hire to do our work. The resulting situation is from top to bottom the agency fails ever worse and costs ever more.
The solution is pretty simple. There needs to be a wholesale cutting from the top of the dead wood of the agencies. We need to fire about 99% of the people in management and start at the bottom rebuilding. The GS system is built the reverse of this. Bumping needs eliminated. GS people need to have manditory retirement at set ages and terms of service. The legislative support of contractors needs changed towards performance controls and away from managing personnel of contractors. Frankly the US agencies excepting where sensative data or methods are involved should have no influence over hiring of contractors. Even then it should only be security issues and not the qualifications at issue. Contractor companies should have to be performing based on results and paid accordingly.
Had the FBI contract had a penalty of $250,000 per day for failure to perform the results would be in hand and done now. This by the way is typical contracting rules in the civilian arena. I have worked on such civilian contract rules for years. It gets work done and on time.
As it sits it is typical for contracts to demand 10 years experience in .net. (Programmers will have to laugh their heads off on this one. .Net isn't that old itself)
Re:Government Inefficiancy (Score:3, Insightful)
Instead, I don't see why companies aren't fined (put it in the contract) or sued for everything the government spent on a system that has to be scrapped. Smaller companies would be run out of business, but rightfully so, and even big companies would have put all that work into it for absolutely nothing. Those are both pretty good motivations to get it right. But that probably doesn't happen because, as someone said, there's no one person ultimately responsible for the money, and those government employees who deal directly with the multibillion dollar contractors may hope to get a VERY lucrative job with them eventually, and so stay cozy with them.
That wouldn't surprise me at all. (Score:3, Insightful)
Low bids the root of all government screwups (Score:3, Insightful)
Look at pretty much any government building that was built on the lowest bidder system. I can pretty much guarantee it has mold or leaking issues.
Going paperless (Score:4, Insightful)
For example, our housing lottery system was, until this past year, an in-person process where people were assigned times, showed up, claimed rooms, and was a fair system that worked. Then, the university got all fancy pants and replaced that lottery with this unbelievably crappy system called Residential Management System [rms-inc.com]. To use: kill ad blocker, only use it in IE for Windows, ensure javascript settings are correct, and then wait until the clock allows you into the online lottery system. Attempt to use a non-intuitive UI that is completely new because you couldn't look at it before while time ticks away and other people claim the rooms you wanted. Even though I got the room I wanted, the experience was horrifyingly bad.
For these large organizations, I think less can be more. Keep your paper trail, but create a highly efficient system for digitizing documents. That way, you start to have some advantages of computers (search, organization, cross-referencing) without the liability of a completely paperless system. From here, you can slowly make a transition from leaning on paper to leaning on machines. But that would be the sane way of doing things, and we're talking about a governement organization here.
Re:Government Inefficiancy (Score:5, Insightful)
On time
At, under, or near budget
Performed as designed.
Mark it flamebait or troll if you want, or just reply with any example.
Insanity (Score:5, Insightful)
Re:Government Inefficiancy (Score:3, Insightful)
He's got it all backwards!
Typical of Large Projects (Score:5, Insightful)
The project has to be bitten in chunks. Lay out the functionality, and then start implementing it one small piece at a time, integrating as you go along. The Big Bang approach is always doomed to failure, or explosive costs, especially when you get to the reality that to deploy you need to shut down the business for two weeks to manage the data conversion. Lot's of small $1 million projects are more likely to succeed and be at budget then one big $20 million project.
This isn't news. It's the whole momentum behind a lot of modern development techniques such as Agile, or architectural such as SOA.
There's also a corrolary that any project involving a big consulting company like EDS, CSC, Anderson(or whatever the hell their name isnow), etc. is more than likely going to cost double what it should.
Re:What world does this guy live in? (Score:5, Insightful)
Seconded. Clearly this guy either doesn't know what he is talking about or is just playing politics (office and/or party). I've personally encountered bugs or (incomplete features) in past releases of Oracle. I don't recall the specifics of the feature I was trying to use, but it was a documented feature that should have been available and according to Oracle's own knowledge base the function should have worked a certain way, but only to dig a little deeper to find that it was just a stub function that hadn't actually been written yet. This was an enterprise product used by thousands of big businesses and it simply didn't do what they said it did.
To say that you are just changing colors on a software product a month before delivery is a rediculous thing to say, and really this guy shouldn't be in his job if he actually believes what he said, vendors are working on bugs for years after delivery on anything as complex as this would need to be.
Hell, even NASA even built in a way to update the software on the Mars landers, when they were on Mars. That isn't to say that this FBI software project has been well managed, well specified, or even well coded, but a certain amount of imperfection must be understood in any project management and design.
Re:Most IT Projects Fail (Score:1, Insightful)
You misunderstand: allow me to translate:
They want a particular person. That's why the specificity. You need not apply.
Sometimes the problem is the specs. (Score:4, Insightful)
I'm not going to criticise the folks who were trying to implement the system until I know a lot more about the actual conditions in which they were trying to work...
Re:Government Inefficiancy (Score:4, Insightful)
That's true for large corporations as well. I worked as a contractor in the Chicago area (not for Anderson Consulting), and had one customer (pharma) tell me that they brought in Anderson for $3m/month for over 2 years, and got 'nothing' for their expense. I saw the same thing with other companies, including one of the largest building-controls companies in the world. It seems that size is the killer, and the reason that federal govt projects are so expensive and delayed is because of the size (not the nature) of the organization.
Re:Government Inefficiancy (Score:2, Insightful)
Re:Government Inefficiancy (Score:5, Insightful)
The driving problem is the rigidness and stagnation of the government's bureaucracy. The impulse to build this kind of lumbering bureaucracy was a good one - it's called civil service, and it's basically a way to insulate long-term government functions from short-term politicians, keep government employees from becoming the minions of whichever politician wants to build a personal empire. There's no question that the limitations of that approach are killing government. On the other hand, do you really want a civil service that can be downsized or force to work on producing bogus intelligence so we can invade 17 more countries? Or a government whose job is to buy as many copies of Microsoft Vapor Server as it can possibly cram into an appropriations bill? The idea that the government is fundamentally incapable is a useless one, sorta like existentialism, in that it fails to answer the question of what we DO as a result of that insight. Do you seriously propose that the FBI be run by a private interest? I'd rather not have someone like Verizon or ChoicePoint watching my back, thank you - at least the government has a mandate to protect people and not just make money off of them. And there's a name for the head of an large private armed force: warlord.
The article touches on the fact that government has progressively become a comparatively worse place to work than the private sector, because of the bureaucracy and because the salaries don't keep pace with the private sector. A friend of mine is working on Sentinel, and he's been really surprised to find an FBI-side partner who actually wants to oversee the work. If you do think that police work at the federal level should be the job of government, then how do we go about really fixing the FBI?
Government is what citizens make it. And here's the rub: under the past 25 years of leadership of the small-government zealots, we managed to prevent government from making important investments - e.g.: roads (any idea how many bridges in this country haven't been maintained in decades, and what the long-term maintenance will cost on the vast numbers of roads we've built?), emergency planning, a healthy population, an educated workforce, etc. These investments are the infrastructure on which the economy is built. And this stellar leadership has not only managed to give short shrift to the future, but it's utterly failed to address the real problems they correctly identified with government. Anti-government conservatism is a bankrupt ideology - it's nice to kick the government for it's failures real and perceived, but when push comes to shove, it offers no real alternative for building the public underpinnings of our economy and our lives, just faith that the free market fairy will come fix all our problems. We live in an extraordinarily pragmatic age: one where you can assemble data on a large scale to decide if something works or doesn't. It's time to stop carping and give our government a mandate to do this and find its way out of quandaries like the civil service vs. Tamany Hall problem.
Sorry for the rant. Somebody talk to me about fixing the FBI.
Re:Project Managers (Score:2, Insightful)
her: have you completed X?
me: not yet, but I have Y and Z completed and tested.
her: but X is on the schedule to be complete today.
me: It made more sense to do Y and Z first, X will be trivial now that Y and Z are complete.
her: but X is on the schedule to be compete today. Y and Z are not scheduled to be done until next month. now we are behind schedule.
honestly, I can't make this stuff up. She actually said that as I stood there in slack-jawed amazement.
Re:Government Inefficiancy (Score:2, Insightful)
Right. Like the government builds its own F-15's, right? Really, different organizations are good at different things. The FBI is an investigation organization, not software development. Their expertise is (or should be) investigation and they should outsource those things that are not in the area of expertise. Just like I don't build my own house, I pay someone to do that that knows more about it than I do.
What amazes me is that projects like this can "fail." While it's not within the FBI's area of expertise to develop a computer system, it's not rocket science for those in the field. Heck, give me half that--$85 million--and I'll develop the friggin' system myself.
Small potatoes (Score:2, Insightful)
730K for $170 million? For the government, that is nothing. It is actually a pretty good price for that amount of code. It also seems like it was a pretty quick project too. And hundreds of SPRs a week before release? Not bad!
From my government contracting experience, none of this sounds that bad. Hopefully there is much more to the story that they aren't talking about. But from the examples they are using, SAIC's performance sounds distinctly "above average." It may have been a disaster, but TFA does not give us enough accurate information make that judgement.
Re:Government Inefficiancy (Score:3, Insightful)
I have to agree with this. I've worked as a contractor along side civil servants and uniformed military personnel. It's rare for the military folks to stick around more than two years in one place, and one job - in some posts, 9 months was about average. The civil service guys had typically been around for many years, and had a much greater sense of ownership of their systems and processes.
I worked on a system that was developed in-house by civil servants. It was an effective system because it was developed by the people who needed it, and it was relatively inexpensive. I worked on it as a contractor, but I was there long-term (most of a decade), and it was MY system - I was right down the hall from the guys using it, I spent enough time with them to know at least the basics of all of their jobs, and it was MY problem to make it all work. I had a vested interest in making sure it ran smoothly and required as little maintenance as possible. After all, less time fixing things means more time on Slashdot.
But along came a sweeping billion-dollar modernization project, and someone decided it was time to replace the system. That was around ten years ago. Tens of millions of dollars later, they still haven't matched the usability of the old system. But the contractors have no incentive to make it work. Once they hand it over, it's ANOTHER company's job to maintain it and fix it. There's way too much separation between the people doing the work, and the people using the end result.
complex or simple? (Score:3, Insightful)
But that would be cheap and quick to implement and not much chance of making a vast profit.
Re:Government Inefficiancy (Score:3, Insightful)
They don't know what they want (Score:5, Insightful)
You'd probably think so, but I bet after the first few months of totally contradictory change requests, specification creep, and an utter lack of hard-and-fast acceptance criteria, that you'd throw up your hands, too.
You can blame the contractors all you want, but I've worked on a bunch of projects like this, and they almost always fail not because the developers weren't good or didn't know their stuff, but because there wasn't somebody on the client side who had the political (internal/office-politics, not Democrats/Republicans politics, although within the USG they're often related) capital to get all the little fiefdoms that exist inside a big organization and sit them down and say "Okay, Fuckheads: this is the system we're going to be using, this is how it's going to work, and you will use it."
Projects like this fail when you let every Tom, Dick and Harry start pushing features into it. I've seen situations where software is in the final stages of testing, and somebody decides that it would be fun to bring down the Big Boss to show them where all these millions of dollars have been spent. And the Boss will come down and take one look at the software, and immediately demand that something get changed. Often I don't think that they really care about what they're demanding, they just want to show off that they have the power to change shit, so they do.
It's stuff like that which pushes projects into failure, even if they look dead simple on paper. The problem isn't a software-engineering one, it's a customer-relations one. It's a problem of the people hiring the developers probably not having a good idea of what they wanted in software, and not having a single person in charge of it.
You can tell that happened with this FBI project, because it's obvious just from the summary that the CIO wasn't involved in the project throughout its lifecycle. He just seemingly walked in on it when it was a month away from deployment, at which point I'm sure everything was totally FUBAR. The way to have prevented this would have been to get somebody like that on board from the very beginning, who could have kicked ass and taken names and kept things under control.
Without good leadership on the client side, and a clear set of business processes, requirements, and acceptance criteria, it's not surprising that these large software projects fail as often as they do. However, as long as the failures are equally profitable to the development contracting companies as the successes, they have no problem taking on a contract even though they know the client is going to drive it into the ground and has no idea what they want.
They just don't get it... (Score:1, Insightful)
I can't imagine something like this costing over $100 million dollars. Yes, I realize the importance, complexity, and scale, of the system but, still, $100 million? No way! Just another example of what should be considered criminal activity by a company: bilking the government for all it can just because they can. Such a system could be easily developed for under $5 million dollars and in a year or less by a good, solid team. Cut all the bureaucracy out, focus on the customer (the FBI), and just *work*.
This system could be a vital tool in the war on terror and in crime management in general. Contractors need to take this stuff seriously. It's about more than *just* money. It's about national defense and homeland security (yes, I know, I know). I suppose I'm ranting because I am just tired of seeing this absolute incompetence run rampant through government and I'm tired of watching company after company steal taxpayer dollars by exploiting that incompetence.
I'm sending a letter to Director Mueller and Mr. Amzi today offering a team to do the work for a very fair and affordable price. My team won't be motivated by greed or dollar sign filled eyes but rather by a sense of national pride and a strong desire to really help in the war on crime.
I just wish more companies acted on those things. Revamping all these agencies would be a piece of cake and a LOT more affordable.
-- Anonymous
Re:Government Inefficiancy (Score:3, Insightful)
Really, it goes far beyond this (not that I don't fundamentally agree with your point). SAIC is simply another war profiteer and 9/11/01 "security" profiteer aligned with this administration. Their profits have soared with the attacks of 9/11/01 and the invasion and bloody occupation of Iraq. One need only look at their personnel roster to get a solid impression of what's wrong with the present fascist regime and globalist congress.
The point is a transfer of wealth (taxpayers' monies) along with money laundering, it has absolutely nothing to do with achieving any goals. Certainly nothing to do with national security - as if anyone in this administration could pull their noses out of their assets portfolios long enough to notice....
Re:Government Inefficiancy (Score:1, Insightful)
E is actually the only point with any validity, and is a major one.
Re:And this is different from the norm because? (Score:4, Insightful)
Although this is true, it is also true that most programmers deliver crap looking, uncomfortable to use, half-assed interfaces if left to their own devices. I know I do...
The same is true of a lot of Open Source software. Fun to do, very powerful, but much of it does look unprofessional or at least unreasonably hard to use, except for other programmers who share the same mindset as the maker.
The tragedy is on one hand that the people who complain about the interface issues are themselves also totally untrained and unqualified to say what exactly needs to be changed, and on the other hand that of course a solid, great looking interface design should be made up front, in the design phase, by professionals. I don't think that ever happens.
But we programmers can start by looking a bit more critically at our own work. A bit. While bitching about those irritating users who think looking professional matters more than actual function. Right?
Re:Project Managers (Score:1, Insightful)
Re:Government Inefficiancy (Score:3, Insightful)
1) First step, IDENTIFY THE REQUIREMENTS of a new system. Create use cases. Observe what actually is done day to day for at least three to six months (this will need people with security clearance out the wazoo). Be sure to follow some issues beginning to end. Also, identify relevant policy and law that the FBI needs to observe - the system should be build to help ensure that proper steps are taken, both by making it easy to do the mundane work and hard to do things that will have a big negative impact on people (make the latter tracable to individuals in ALL cases - build accountablity into the system.) And for goodness sakes, don't throw out ideas in the old system if they are good, just because they're part of the "old way".
2) Second step, design and implement a small scale trial in one or two offices, working in parallel with the existing system. Identify and correct problems.
3) Gradually scale up, and in each new introduction have people familiar with both the FBI's old system and the workings of the new help with the transition. Be resigned to huge amounts of grunt work, figuring out where old files are, cataloging and re-entering info, etc. etc. etc. Because this is a matter of law enforcement, all old materials should be retained in case of mistakes. Budget for all this annoying, unsexy, but essential work. The framework must be strong enough to handle what will be put in it, but putting the content into it will be a huge task and that should be part of the design stage.
I personally think this is one situation where things like provably correct software are needed - law enforcement doesn't need any more problems, and lawyers don't need to get a chance to play around with "computer glitch" gotchas or problems.
Large systems are often hopelessly overweight and complex, so they should not be involved with the technical design. Such organizations ARE, however, very good at following regulations, instructions, and systems. Those traits should be utilized as much as possible.
One other design feature should be that all records, when updated, should have a printed record be automatically generated and stored somewhere (or several places). Computer data is too easy to change - the poster who mentioned tracibility is absolutely correct. A hard copy is MUCH easier to work with, and automatic organized printed records should be a part of the design from the get go.
Re:Um, yeah, it's called "matching" (Score:5, Insightful)
They just "matched" it. That's the industry term. As a stringer for many years (a "stringer" is a type of freelance journalist) I was called by editors many, many times to "match" stories.
You've worked in journalism for, what, a week now? Welcome to the industry. You may want to check with some people in your organisation who've been around the block a few times before firing off embarrassing (to you) letters to the Post Ombudsman.
You help me understand why the mainstream press is in such bad shape these days. Shoddy Lazy Journalism is accepted as standard industry practice.
Third time's a charm (Score:2, Insightful)
Every Fortune 1000 company has hurtled down this obstacle course and has the scars to prove it. It is an expensive journey, but there doesn't seem to be any way around it. As some say, "Pay now or pay later." Multiple iterations of large projects are the norm, not the exception and I believe are an inevitable learning curve that can't be avoided - a rite of passage so to speak. The only disconnect in the FBI's project was thinking that it could be done right the first time at any cost or in any timeframe.
Why? Because half the issues are cultural and derive from the dynamics of the organization which needed to change after 9/11. No vendor or project manager or JAD team can solve for that - they were caught in the middle of a paradigm and culture shift that they had no control over and may not even have been fully aware of or able to articulate and document. The fact that their JAD sessions lasted 6 months is surely proof enough that the organization wasn't ready to talk to IT developers.
Setting aside the lapses in personal competence (e.g. great effort to collect trinkets and souvenirs for scrapbooks while putting a value judgement on valid criticism and calling it "disruptive"), at the end of the day, you can't design what you can't conceive. They didn't know what they didn't know. Now, somewhat older and wiser, perhaps they know quite a bit more, but I doubt they (the FBI organization) know enough to get it right.
One of the "graybeards" in the IEEE article predicts it will be 2010 or 2011. But that will only be the second time. I say it will be more like 2015 for a third try before they will really know with any confidence what they actually knew about the hijackers before 9/11 and have enterprise application systems that are world class and interoperable both within the FBI and externally with other intelligence and law enforcement agencies.