Last night, I had the privilege of attending a grip-and-grin with the brains behind the Obama campaign’s information technology strategy. And I took some notes:
- The Obama campaign enlisted people with deep, proven IT and engineering backgrounds. “None of us are from the [political] campaign culture,” said one panelist. “We all have engineering backgrounds.” There simply were no junior-level staff writing code or making IT decisions.
- By contrast, the Romney campaign hired expensive consultants and did little to manage their work. One panelist estimated the Obama campaign spent around $500,000 on IT while the Romney campaign spent $15m to build similar infrastructure. “But, you know, I’m sure they did it right,” he said.
- They built both the platform and the apps with open-source technology and deployed them iteratively, with an eye towards scalability. At some points — right before the DNC, I believe, and again after Obama affirmed his support of marriage equality — the infrastructure was processing $2m an hour in donations.
- Another panelist likened the campaign’s technology to a one-billion dollar, “disposable” startup that simply vanished after 18 months, which struck me as a great description. They built the apps and the underlying platform at the same time, apparently.
- "We logged and measured everything," said another panelist. "I can’t imagine making decisions in any other way."
- At the same time, they used direct user feedback to tweak the apps and the underlying platform. “If it doesn’t matter to the user, you throw that shit out,” one said.
- Their platform used Amazon SQS to queue messages. A billion messages cost the campaign just a thousand bucks.
- Someone else said: “We weren’t building Narwhal [the Obama campaign’s IT platform] for performance, until we tried importing all the history of voting, ever.” I think this was a joke.
- Apparently, the campaign was using Google Docs to hold and manage so much data that they nearly crashed it.
- They used something called a “sentient chaos monkey" to test redundancy and disaster scenarios. As Hurricane Sandy approached the eastern seaboard, the team moved everything from the East coast to the West coast in less than 24 hours. And the team worked on hypothetical disaster scenarios even as real-world problems with the technology arose.
(Image via Caroline Waxler’s Twitter. Thanks to New Relic for hosting.)