We Are Currently Experiencing Some Network Issues
I’m sure that many of you have already figured out that we have been experiencing some network issues tonight. We’re sorry about the service interruption, and are working with our network providers to rectify the issue. At this time it appears that some routine maintenance has gone south in a big way in one of our locations. Unfortunately due to which location is having the problem we’ve had to put some services into read-only mode, and other services have experienced degraded performance. Bear with us, and we will have the issue resolved as soon as possible, and restore everything to working condition.
Yes certainly! I noticed it and so did people on twitter and elsewhere. The pingdom people wrote on their blog that WordPress was down for just 20 minutes from 10 Nov 2008 to 10 March 2009. But of late the outages have become less uncommon!
Just an update on the issues. First – everything should now be back to normal. I have a bit of information on what happened, but am still waiting on a full investigation from our network provider to determine the exact cause of the problems.
At approximately 11:05 PM CST our monitoring systems notified us of a widespread network outage in our Dallas datacenter. This is our largest facility where WordPress.com is hosted. A few minutes later, we had confirmed that the entire datacenter was inaccessible because of a network outage. Although, WordPress.com is served in real-time from 3 datacenters, there is quite a bit of communication between locations to keep everything in sync. We immediately took steps to implement a “read-only” mode which allows us to serve blog pages from our other facilities while we further diagnose and work around the network issues. This was complete at about 11:20 PM.
At approximately 11:30 PM we began to see a partial recovery of the network and were hopeful that the rest would soon follow. Unfortunately, it took another 90 minutes for some critical pieces of the network to come back online. After working closely with our network provider to ensure the stability of the network, we re-enabled writes to the site and everything was back to normal around 1:15 AM CST.
We take uptime and stability seriously, striving for 100% uptime and are disappointed in the fact that 2 very similar outages in the same facility have caused downtime recently. Unfortunately, the pieces of the network which have failed are not in our direct control, but we will explore what options we have to change that or take other steps to make sure this doesn’t happen again.
Again, I apologize for the downtime and inconvenience it has caused.
Actually, guys – that was a really fast recovery time – and just fyi, the Tweets came right in to announce your return. Thank you for posting the info for us and getting your adoring fans blogging so fast again!
Exhaustive explanation Barry! 100 per cent uptime will remain a chimera for any company but to strive for it is a noble aim. It is 99.99 per cent just the same.
Thanks Barry, for the informative update.
Phew! Local news was boring; so, went to check my 10PM sked post – 11PM CDT, 10PM CST – and the 6PM post was there. And unable to complete login. But, there was a message from the bridge.
Checked Om’s sites and they were passing through the same problems – knew then it was a system-wide problem.
Went to bed.
Everything’s fine this morning. Only discrepancy I can see is sitemeter must have kept on recording hit while WP.com internals didn’t.
Usual diff between the two is less than 5%. This morning, the difference recorded for yesterday is more like 15-20%.
Thanks for doing a great job. Ain’t nothing wrong with being as reliable as Google. :)
I still cannot access my dashboard or view my blog. I’m obviously logged in but that’s as far as I can go. I’ve tried clicking on numerous links to work around the direct route. Nothing works. My browser just shows “loading” and never goes anywhere. My connection is fine, everything else is working but WP.
Is Barry or someone from WP support out there?
Hmm… This is still down here. I can log in, but I cannot edit anything –it just show blank. ISP etc is fine. Any idea when it will be fixed?
Correction: the only thing I cannot edit are the posts. Which is of course kind of what you want to do…
i’ve noticed something weird. I’ve had a very sudden increase in visits with most of the views being on one particular image. i’m at http://tonypritchard.wordpress.com/
This is still not working. Since the outage, I can no longer edit a single post. Help!!! I am at http://coudert.wordpress.com/, what is wrong?
Help me, please!
I can not edit or write new post since two days ago
What’ s wrong and how to correct it?
My site is completely down and the timing of this could not have been worse since I just switched from blogger to here using my custom domain.
I can access my dashboard but I, and everybody else, just gets a cannot find server message when trying to access http://www.lakingsnews.com.
Can someone help with this? We have a decent following and I don’t want to lose readers.
The site was up for an hour and went down again.
So, is this the level of service I can expect from wordpress? I hope this is a fluke and I didn’t make a mistake bringing the blog here.
lakingsnews – this had nothing to do with us. It knocked how many many sites, not just wordpress.com and was entirely out of our control. Barry and others worked through the night to get things back to normal.
Fact: The state of Dallas could disappear and we would get your blog working. One of the other states could disappear and we would STILL get your blog working. Find any webhost that uses 3 datacenters and that guarantees what we do.
You are of course free to move anywhere at any time.
I checked again and even though posts can be viewed, I still cannot edit any post or add new ones. Do you know when we can expect to have the site back to a fully operational state? Thanks!
I think we have same problem, although we shout here anytime, but we have no explanation and anwers clearly :( :( :( :( :(
Umm guys, please read the post above by Barry, and understand that WordPress, like all other entities on the web, cannot be expected to be totally glitch-free. The good thing is, the guys behind the wires are really up to it. So be patient if something like this happens. I know that’s hard to do with WP: it’s so perfect, you don’t expect downtimes. But don’t worry, we’re blogging with the best.
@calexilla – I fixed your blog — there was a problem un-related to these network issues which was causing it to load slowly.
@coudert – Your blog works fine for me. I am able to edit posts without a problem.
@akhmadsudrajat – Same thing. Does the problem happen when editing any posts or just a specific one?
@lakingsnews – Your site seems to be working fine. DNS changes sometimes take time to propagate (24-48 hours). It worked from every place I tried which was 3 points in the US and from Canada. If you are still having problems, please let me know.
The topic ‘We Are Currently Experiencing Some Network Issues’ is closed to new replies.