Hey folks, we have everything back online and all of your data verified, but we’re going to be keeping blogs offline for a while longer while we re-synchronize the backup systems and take the problem hardware out of rotation. The blogs will come back online in waves, about 25% at a time. Things will be back to normal in just a little while longer now.
Thanks to everyone for their patience, and especially to drmike for helping out around the forums.
Also, we have a pretty significant hardware expansion already underway where we’re doubling the number of machines serving WordPress.com, meaning a lot better performance and redundancy. We’re bringing an additional datacenter online in Dallas in addition to our current one in San Diego.
We’ve had hardware failures in the past, as any large web service will experience, but we’re set up in a way they only should cause slowdowns, not downtime. However this Sunday there was a window when things weren’t fully replicated (meaning at least 2 live copies of the data at all times) and that’s when the problem hit.
Once the second datacenter is online even a catastrophic failure, such as California falling into the ocean, would not take WordPress.com down.
I’ll be blogging more about the above when it’s ready and live, and for now I’m going to get back to getting everything running rather than posting here.
Thanks, Matt, much appreciated. Just take the time you need to do things right…
Cant wait to see my blog back.:)
I just got my blog back and I successfully logged in. And then it says the active theme is broken. I chose the theme and viewed my site, everything was okay. A few minutes later my blog was back to the “Update!” and “downtime” page. Can anyone explain this situation?
Just hang on a little longer; there are bound to be hiccups after something as major as this.
Same here. It was back for a couple of minutes but looked so ugly and messy, i’m almost glad it’s been replaced by the message: “However we are going to keep blogs offline for a few more minutes”.
It’s been down for over 12 hours now. As to hanging a little longer, I’m bracing. When it’s back online, I’ll be busy sending emails to dozens of people and updating links to tell people “sorry”. I’m on the job since Saturday evening and running out of coffee, Mum has gone back home a LONG time ago. :(
Well, it was pointed out by someone on the mega-thread (cornell, I think) that recovering from this kind of failure takes at least nine hours, so best just to think of them as really long minutes.
Some blogs will be up, some will be down, we’re very close to getting everything 100%.
Thanks, Matt. I imagine it is a frustrating day for you. I’m glad I was able to get some updates here.
One thing you might change in the future is your error message. It’s been saying “you’ll be back up in a few minutes” for hours now. Always underpromise and overdeliver. It makes us a lot less frustrated.
Everything should be back up now for all blogs. Sorry for the false starts earlier, I know it was frustrating for some folks to see their blogs for a few minutes, and as Robert pointed out the error message stayed for far longer than we would have liked.
We’re going to be tracking issues through the feedback form, so if there is anything wrong with your blog, please let us know there instead of in the forums. Thanks again guys for your patience today, it was certainly an interesting Easter! My girlfriend Glenda managed to break or injure her toe as well, so I’m off to the hospital to get that looked at.
Hey thank you for all the hard work and congratulations on that recent venture capital!
Thanks for all the hard work. I, personally, am grateful for all that you folks do. I owe you a beer when I see you — just make sure I know its you…
Ya done good, guys! Thanks for all the hard work and the patience you showed with us.
my blog is back… thanks Matt, and I must add you guys handled this well. Its a big deal having catastrophic failures, and you’ll have them again. Well done!
Thanks for working so hard on an Easter Sunday, Matt, and I hope your sweetheart’s toe gets better soon!
Thanks. All is working and it looks great.
Looks like you got things back online. Thanks, Matt, for the service you provide and all the hard work. I’m still lovin’ WordPress.
I decided to go to church today and spend the afternoon with my family instead of blogging! Wow! Probably a good idea anyway, right?
Hey, we’re back up and running. Yippee. Bummer for Matt, though. Thanks for all your hard work keeping all our words alive.
My blog is back, but I’m still missing my posts from Friday night. Are they going to be restored as well?
The topic ‘Downtime Update’ is closed to new replies.