Wednesday, May 3

Our brand new "world class" data center ...

... lost power this morning for the better part of an hour.

Five nines? Nah, that's just for those private indusrty pukes. We're government. We don' need no stinkin' reliability!

There was even a cherry on top of this irony sundae - the operations staff down in Salem had to call us up here and ask us to answer their hotlines until the power came back on.

Gee. What was that you said to us a few months back?
Mr. SDC Manager Man: "We don't need a Portland office. We want everyone sitting together in Salem."
Me: "But what about geographic rudundancy? What happens if something takes out the Salem facility?"
Mr. SDC Manager Man: " ... " [insert 30 second pause here] "What?"


  1. Pragmatism dictages that it's time to re-evaluate the single location idea.

    One of the weird things about working at a place that spends public funds is you start to question the necessity of things like redundency. It could really save money....but....

  2. Careful, Andy. "Redundancy" is bad... even when it's technically correct.

    "Failover," on the other hand, is good.

    And my personal favorite: "Either we start building fault-tolerant systems, or you start hiring fault-tolerant people." ;-)

  3. Public funds or not, building fault tolerance and redundancy into IT systems *always* pays off in the long run. It's a lot cheaper to buy two of something than not have a spare when you need it.

    But right now I'm just not feeling particularly civic-minded. I'm feeling just plain petty and vindictive. Should I taunt them about the stellar reliability the new facility is demonstrating? Or perhaps I should just mock them in return for all the times they've mocked us with, "So when y'all moving to Salem?" Or should I just send a two word note to the idiots in the suits that says, "Got redundancy?"

  4. "It's a lot cheaper to buy two of something than not have a spare when you need it."

    We sure learned that where I work last week. The server's power supply went out overnight and we didn't have a backup server. The only thing I could do was check my email and work with my print-out from the day before. Funny how we now have a backup server. . .