Also, we have a pretty significant hardware expansion already underway where we're doubling the number of machines serving WordPress.com, meaning a lot better performance and redundancy. We're bringing an additional datacenter online in Dallas in addition to our current one in San Diego.
We've had hardware failures in the past, as any large web service will experience, but we're set up in a way they only should cause slowdowns, not downtime. However this Sunday there was a window when things weren't fully replicated (meaning at least 2 live copies of the data at all times) and that's when the problem hit.
Once the second datacenter is online even a catastrophic failure, such as California falling into the ocean, would not take WordPress.com down.
I'll be blogging more about the above when it's ready and live, and for now I'm going to get back to getting everything running rather than posting here.