Notes From Dot Gov Land

Economics Internet Culture Memes Nature News Psychology Technology

As some of you may remember, I spent 2009 writing a memoir about my experiences in New York City's "new media" industry from 1993 to 2003. I've often wondered if I would ever write an update.

I might someday, and I might even write about the work I've been doing since 2009, when I moved down to Northern Virginia to get married and began working in Washington DC and in Northern Virginia's tech corridor.

I only write memoirs in past tense, so I won't be writing about my current jobs and projects anytime soon. But I wish I could, because lately it's been as exciting as Silicon Alley down in here. The big local story is the epic #fail of the Obama administration's website, which was built by several NoVa firms like CGI Federal.

The website's crash has embarrassed this region's entire tech community, especially since a few uninformed commentators have speculated that DC/Northern Virginia doesn't have the tech skills to build a high-capacity website, and that Obama ought to jet in a team of Silicon Valley hotshots to fix

This is really off base, especially since we build and operate plenty of high capacity websites down here. Let's see, there was this little outfit called America Online that handled a bit of traffic out in Fairfax. There's the Pentagon and the FBI and the CIA and the Library of Congress -- a couple of demanding customers in town. We have TCP-IP guru Vinton Cerf working in the local Google office in Reston … so, yeah, I think we know a little bit about how to operate high capacity websites down here in DC/NoVa.

So why has been such an epic failure? Well, first, they didn't build the site with open source Drupal software. Terrible decision. Drupal is the most popular web publishing software in the federal government community. (In fact, I only learned Drupal myself in 2009 because it was required at the first job I got after moving down here).

Drupal is free and open source, and it's a very common choice for new federal government websites. It's also particularly great for custom user account functionality, which is what is all about. So I really can't understand why the architects of decided to go outside the obvious open source software choice for their website. Probably some executive was trying to "think outside the box".

Or maybe they didn't want to use free/open source software at all. When I first heard about the debacle, I didn't comprehend the dimensions of the project, and I felt confused why CGI Federal couldn't have built and tested the website correctly. Then I heard the shocking news that the government paid CGI Federal around $90 million dollars to build the website, and then I understood the problem. If they had only paid $5 million dollars, the website would used open source software, and it would have worked. But if you pay $90 million dollars, the big shot wheeler-dealers and corporate snakes get involved, and the money starts to become more important than the website. A gigantic budget can really be fatal to a software project (this was the same lesson I learned while working for Time Warner, as discussed in my Silicon Alley memoir above …)

Because I've worked in many industries from media to finance to government, I also don't buy the idea that the federal government is any worse at running a website than any other high-flying business sector. We're all terrible at it, and every website launch ever since the beginning of time (okay, the beginning of 1995) has been a stumbling mess. The crashes just don't usually get this kind of attention.

Especially at the lofty levels when big money gets involved, the federal government's tech bureaucracy is exactly like Silicon Valley's or Wall Street's or Madison Avenue's. I'm there, and I see it. It's always the same cast of characters, the same ecosystem, the same bloated egos in the boardrooms, the same servile obsequiousness in middle management, the same Aspergers syndrome in the cubicles.

I am writing this blog post on November 26, four days before Barack Obama's self-imposed deadline of November 30 for the newly fixed In all seriousness, I pray that the website will stop crashing -- and I know that far more than my local Northern Virginia techie pride is at stake. The success of health insurance reform in the USA is also at stake, and that's a success we really need -- for our health.

12 Responses to "Notes From Dot Gov Land"

by Jody Cline on

Thanks, very interesting to a (former) techie. What development platform did they use & do you know or can you guess why using that technology caused the failure?

by Levi Asher on

Hi Jody -- well, there are many components using a variety of platforms. I can't detect evidence of a single universal platform and that's part of the problem, because Drupal really demands that you do everything "the Drupal way", which takes a lot of the uncertainty out of the engineering. It's a tried-and-true platform that can always get the job done.

When I look at this site, I see some Ruby but I really can't tell for sure what the main platform is. If the main platform were Drupal, it would be very easy to determine this from looking at the source. Does anybody else know?

by Jody Cline on

I'm just curious... No "need" to know....Your article inspired me to do some further reading and there are many theories out there about what went wrong but the most prevalent seems to be a design the sends loads of data to the user causing a cascade of errors to follow. Thanks for the inspiration to read further!

by Bill_Ectric on

This is the first sensible explanation I've heard about the problem.

by Bud Parr on

There are two parts of the site. The front-end marketing site and the backend application where users pick their plans, etc.

The backend is where the problems are and Drupal or any CMS would not have been even close to appropriate for that. The problem is not with a software choice at all.

The front-end is indeed open source. In fact, it's...HTML! You can find an exact replica of it on github and run it on your local computer (I have). It was created with open source software called Jekyll, which generates static HTML pages. The shop who built it used to be a Drupal shop and have completely abandoned that for using Jekyll. You'll note that there are no problems with that part of the site.

This technique of creating static pages was also used for Obama's successful fundraising during the last campaign. That site has given steam to a whole new movement toward a CMS-less, database-less web (where it's appropriate), and I've been spending an enormous amount of time thinking about it, and it's awesome.

by Levi Asher on

Bud, thanks for the excellent info. I have heard of Jekyll but did not know this was the basis of the site. But I don't think your breakdown into front-end vs. back-end holds here -- there's a third end here, which is "user-end". The first thing you do on is create a user account. Once your user account is created, every page is personalized for you and you are therefore beyond the scope of what a flat static HTML page can deliver. The entire site revolves around the user account. And integrated user-oriented functionality is probably Drupal's single biggest strength.

by Bud Parr on

The "user-end" is part of the back-end application.

by Levi Asher on

Well, Bud, do you know what they used to handle the user account functionality? Jekyll doesn't seem to offer anything here, as far as I can tell.

I took your "back end" to refer mainly to the various external data sources and repositories and co-respondent websites that the main website had to integrate with, as well as its own back end. I think Drupal has proven its capability in integrating with external websites, as well as its capability for dynamic user-oriented page architecture.

by Bud Parr on

It's just that most people are conflating the two parts of the site. One part is for education (the "front-end") and the other part is for getting insurance (the "back-end"). One doesn't need a user account to read articles and the use of a static site generator provides very high security and low latency, which is why it was chosen for that part of the site. And, for what it's worth. The firm that built that part of the site is in D.C.

I'm only taking issue with your statement that using Drupal would have saved the day. Besides the blinding fandom of your statement, it's simply not true. User accounts are but one part of the application, which purportedly pulls in data from government agencies around the country. It's that application that has failed and it would have failed even if they used an off-the-shelf CMS, no matter how wonderful, for user accounts.

Here's a post from the firm that built the marketing site with some links that might give you some more clarity.

by Levi Asher on

Thanks, Bud. I'm all for Jekyll, and I'm glad you're pointing us to these articles about it.

As I understand it, the original website (the one that famously crashed on the day it launched) pushed visitors to create user accounts immediately upon arriving on the site. This reduced the site's ability to provide useful static (Jekyll-based) HTML pages. It was quickly determined that the site could do better with more static pages.

It is true that I am an obvious Drupal fan (though not an uncritical one, and I'm really not sure whether or not I'll be getting on the boat for Drupal 8 when it launches in early 2014). But there is a benefit in choosing popular software, regardless of the technical benefits of the software itself. Because Drupal is so popular and widely used in DC and Northern Virginia, there is therefore a gigantic and helpful Drupal expert user community in DC and NoVa. There are also several major Drupal development firms like Acquia and PhaseTwo with expert staff who could have helped with if the project had been built with Drupal. I guess what I'm saying is, popularity and wide availability of expert skill sets is itself a positive feature of a software platform -- it's one of Drupal's bigger selling points. When it comes to high-risk software development projects, you want to be on a big bandwagon to increase your chances of success. And Drupal is the biggest bandwagon in town for federal government websites.

by Bud Parr on

I don't have a problem with Drupal necessarily (though I don't like it and have tried - another conversation, one that I'm not too interested in having), but fandom as well as the incorrect reporting I've seen of the issue. One major web firm completely missed the mark and wrote an article about how they would have designed the site, providing an alternative design, completely ignoring the fact that nobody had an issue with visual design!

You're right about bandwagons, which is one good reason I'm using Jekyll instead of the myriad other static site generators. It's still relatively obscure because you have to be comfortable on the command line to use it and install Ruby and some other things, but there are seeds of the future there.

But a bandwagon isn't the only reason to use a tool, particularly these days, and unfortunately, that bandwagon also feeds a lot of bloated sites that will have very bad effect on the web.

by ds on

A very interesting insider take. Open source would have been the way to go. I contracted for a big govt department in NOVA for a few years. I hesitate to give more personal details. Really I don't know what I signed when I got the job. But thinking back on it now makes me want to defenestrate.

I think I can say we were a strict .Net shop, and the idea of using open source at least for the site we maintained was unthinkable to the higher-ups, both in my contract agency and this dept. The site is huge, and it wasn't just one contract company maintaining the whole thing. The dept has it split between several contractors, a piece of it maintained by one, a piece by another, etc.

So the trouble there as you can imagine was the inefficiency and the inter-contractor fighting when the dept asked for certain changes. Another problem was the competition between the contractors, each whispering in the govt's ear, trying to win more of the pie slice at the expense of the other contractors.

I would like to tell any powers that be that I'm expressly not blowing any whistles, I'm in a foreign country now, and Happy Thanksgiving! Try to show your kids some love today, you old so-and-so's <3

Add new comment