Downtime Update

  • Author
    Posts
  • #10501

    matt
    Staff

    Hey folks, we have everything back online and all of your data verified, but we’re going to be keeping blogs offline for a while longer while we re-synchronize the backup systems and take the problem hardware out of rotation. The blogs will come back online in waves, about 25% at a time. Things will be back to normal in just a little while longer now.

    Thanks to everyone for their patience, and especially to drmike for helping out around the forums.

    #10636

    matt
    Staff

    Also, we have a pretty significant hardware expansion already underway where we’re doubling the number of machines serving WordPress.com, meaning a lot better performance and redundancy. We’re bringing an additional datacenter online in Dallas in addition to our current one in San Diego.

    We’ve had hardware failures in the past, as any large web service will experience, but we’re set up in a way they only should cause slowdowns, not downtime. However this Sunday there was a window when things weren’t fully replicated (meaning at least 2 live copies of the data at all times) and that’s when the problem hit.

    Once the second datacenter is online even a catastrophic failure, such as California falling into the ocean, would not take WordPress.com down.

    I’ll be blogging more about the above when it’s ready and live, and for now I’m going to get back to getting everything running rather than posting here.

    #10637

    oclcyc
    Member

    Thanks, Matt, much appreciated. Just take the time you need to do things right…

    #10640

    retributions
    Member

    Thanks Matt,

    Cant wait to see my blog back.:)

    #10641

    kenkccc
    Member

    I just got my blog back and I successfully logged in. And then it says the active theme is broken. I chose the theme and viewed my site, everything was okay. A few minutes later my blog was back to the “Update!” and “downtime” page. Can anyone explain this situation?

    #10642

    wank
    Member

    Just hang on a little longer; there are bound to be hiccups after something as major as this.

    #10643

    wrnc
    Member

    Same here. It was back for a couple of minutes but looked so ugly and messy, i’m almost glad it’s been replaced by the message: “However we are going to keep blogs offline for a few more minutes”.
    It’s been down for over 12 hours now. As to hanging a little longer, I’m bracing. When it’s back online, I’ll be busy sending emails to dozens of people and updating links to tell people “sorry”. I’m on the job since Saturday evening and running out of coffee, Mum has gone back home a LONG time ago. :(

    #10644

    wank
    Member

    Well, it was pointed out by someone on the mega-thread (cornell, I think) that recovering from this kind of failure takes at least nine hours, so best just to think of them as really long minutes.

    #10645

    matt
    Staff

    Some blogs will be up, some will be down, we’re very close to getting everything 100%.

    #10646

    scobleizer
    Member

    Thanks, Matt. I imagine it is a frustrating day for you. I’m glad I was able to get some updates here.

    One thing you might change in the future is your error message. It’s been saying “you’ll be back up in a few minutes” for hours now. Always underpromise and overdeliver. It makes us a lot less frustrated.

    #10647

    matt
    Staff

    Everything should be back up now for all blogs. Sorry for the false starts earlier, I know it was frustrating for some folks to see their blogs for a few minutes, and as Robert pointed out the error message stayed for far longer than we would have liked.

    We’re going to be tracking issues through the feedback form, so if there is anything wrong with your blog, please let us know there instead of in the forums. Thanks again guys for your patience today, it was certainly an interesting Easter! My girlfriend Glenda managed to break or injure her toe as well, so I’m off to the hospital to get that looked at.

    #10648

    helpdesk
    Member

    Hey thank you for all the hard work and congratulations on that recent venture capital!

    #10650

    Thanks for all the hard work. I, personally, am grateful for all that you folks do. I owe you a beer when I see you — just make sure I know its you…

    #10651

    Ya done good, guys! Thanks for all the hard work and the patience you showed with us.

    #10652

    bankwatch
    Member

    my blog is back… thanks Matt, and I must add you guys handled this well. Its a big deal having catastrophic failures, and you’ll have them again. Well done!

    #10653

    strambinha
    Member

    Thanks for working so hard on an Easter Sunday, Matt, and I hope your sweetheart’s toe gets better soon!

    #10655

    ninglun
    Member

    Thanks. All is working and it looks great.

    #10656

    pizza
    Member

    Looks like you got things back online. Thanks, Matt, for the service you provide and all the hard work. I’m still lovin’ WordPress.

    I decided to go to church today and spend the afternoon with my family instead of blogging! Wow! Probably a good idea anyway, right?

    Thanks again,
    Tom
    http://pizza.wordpress.com

    #10657

    ptvguy
    Member

    Hey, we’re back up and running. Yippee. Bummer for Matt, though. Thanks for all your hard work keeping all our words alive.

    #10658

    yorker
    Member

    My blog is back, but I’m still missing my posts from Friday night. Are they going to be restored as well?

The topic ‘Downtime Update’ is closed to new replies.