Need help? Check out our Support site, then


WordPress.com Forums » Support

12

Downtime Update

  1. matt
    Staff

    Hey folks, we have everything back online and all of your data verified, but we're going to be keeping blogs offline for a while longer while we re-synchronize the backup systems and take the problem hardware out of rotation. The blogs will come back online in waves, about 25% at a time. Things will be back to normal in just a little while longer now.

    Thanks to everyone for their patience, and especially to drmike for helping out around the forums.

  2. matt
    Staff

    Also, we have a pretty significant hardware expansion already underway where we're doubling the number of machines serving WordPress.com, meaning a lot better performance and redundancy. We're bringing an additional datacenter online in Dallas in addition to our current one in San Diego.

    We've had hardware failures in the past, as any large web service will experience, but we're set up in a way they only should cause slowdowns, not downtime. However this Sunday there was a window when things weren't fully replicated (meaning at least 2 live copies of the data at all times) and that's when the problem hit.

    Once the second datacenter is online even a catastrophic failure, such as California falling into the ocean, would not take WordPress.com down.

    I'll be blogging more about the above when it's ready and live, and for now I'm going to get back to getting everything running rather than posting here.

  3. oclcyc
    Member

    Thanks, Matt, much appreciated. Just take the time you need to do things right...

  4. retributions
    Member

    Thanks Matt,

    Cant wait to see my blog back.:)

  5. kenkccc
    Member

    I just got my blog back and I successfully logged in. And then it says the active theme is broken. I chose the theme and viewed my site, everything was okay. A few minutes later my blog was back to the "Update!" and "downtime" page. Can anyone explain this situation?

  6. wank
    Member

    Just hang on a little longer; there are bound to be hiccups after something as major as this.

  7. wrnc
    Member

    Same here. It was back for a couple of minutes but looked so ugly and messy, i'm almost glad it's been replaced by the message: "However we are going to keep blogs offline for a few more minutes".
    It's been down for over 12 hours now. As to hanging a little longer, I'm bracing. When it's back online, I'll be busy sending emails to dozens of people and updating links to tell people "sorry". I'm on the job since Saturday evening and running out of coffee, Mum has gone back home a LONG time ago. :(

  8. wank
    Member

    Well, it was pointed out by someone on the mega-thread (cornell, I think) that recovering from this kind of failure takes at least nine hours, so best just to think of them as really long minutes.

  9. matt
    Staff

    Some blogs will be up, some will be down, we're very close to getting everything 100%.

  10. scobleizer
    Member

    Thanks, Matt. I imagine it is a frustrating day for you. I'm glad I was able to get some updates here.

    One thing you might change in the future is your error message. It's been saying "you'll be back up in a few minutes" for hours now. Always underpromise and overdeliver. It makes us a lot less frustrated.

  11. matt
    Staff

    Everything should be back up now for all blogs. Sorry for the false starts earlier, I know it was frustrating for some folks to see their blogs for a few minutes, and as Robert pointed out the error message stayed for far longer than we would have liked.

    We're going to be tracking issues through the feedback form, so if there is anything wrong with your blog, please let us know there instead of in the forums. Thanks again guys for your patience today, it was certainly an interesting Easter! My girlfriend Glenda managed to break or injure her toe as well, so I'm off to the hospital to get that looked at.

  12. helpdesk
    Member

    Hey thank you for all the hard work and congratulations on that recent venture capital!

  13. sweetimmolation
    Member

    Thanks for all the hard work. I, personally, am grateful for all that you folks do. I owe you a beer when I see you -- just make sure I know its you...

  14. itsmylifedammit
    Member

    Ya done good, guys! Thanks for all the hard work and the patience you showed with us.

  15. bankwatch
    Member

    my blog is back... thanks Matt, and I must add you guys handled this well. Its a big deal having catastrophic failures, and you'll have them again. Well done!

  16. strambinha
    Member

    Thanks for working so hard on an Easter Sunday, Matt, and I hope your sweetheart's toe gets better soon!

  17. ninglun
    Member

    Thanks. All is working and it looks great.

  18. pizza
    Member

    Looks like you got things back online. Thanks, Matt, for the service you provide and all the hard work. I'm still lovin' WordPress.

    I decided to go to church today and spend the afternoon with my family instead of blogging! Wow! Probably a good idea anyway, right?

    Thanks again,
    Tom
    http://pizza.wordpress.com

  19. ptvguy
    Member

    Hey, we're back up and running. Yippee. Bummer for Matt, though. Thanks for all your hard work keeping all our words alive.

  20. yorker
    Member

    My blog is back, but I'm still missing my posts from Friday night. Are they going to be restored as well?

  21. matt
    Staff

    Yorker, and anyone else who has a glitch with their posts, please send a feedback and we'll get to it ASAP. Make sure to say when the posts are missing from. We won't check your blog out if you just post to the forums, we a feedback request.

  22. burodestruct
    Member

    thanks a lot. keepin' toes and thumbs crossed for your hard eastern time.

  23. chronotron
    Member

    Thanks a lot for working hard even on Easter to bring all this back up Matt. We really appreciate it and I have to bow to you guys for the dedication. Now I can blog in peace! Thanks a lot.

    On a sidenote, I was really annoyed with the member responses!!

    Cheers~
    Chrono Cr@cker

  24. wrnc
    Member

    thanks for the hard work

  25. crossthebreeze
    Member

    Thanks Matt for the hard work, my lost posts from April 13th/14th are still lost though... anything you guys could do about that?

  26. slowleakyblog
    Member

    Thank you thank you thank you--for y'all are jolly good fellows, et cetrea. I'm new to this blogging thang, just recently started one up to keep my widely dispersed extended family informed. Devastated to watch it disappear, relieved to learn I hadn't screwed things up somehow, thrilled to have it all back online. Me like-um WordPress!

  27. isadora
    Member

    Thanks a lot for fixing the problems with the database so quickly! I was very happy to find out my data didn't get lost (never had any doubt about it, actually) and, on the top of it, you even rolled out the Regulus update I was waiting for! A great late-Easter present! Happy Easter to everyone out there and thanks again!

  28. nodependenciesnologo
    Member

    Well done! Thanks for all the hard work. You're the top.

  29. jaotte
    Member

    Glad it's all back - I'm happy you were able to recover everything!

    My side-bar has gone a bit wongely, but maybe messing around with the theme will fix it.

  30. devilcatz
    Member

    greatest`~~~

Topic Closed

This topic has been closed to new replies.

About this Topic

Tags