To make matters worse, during a large part of this outage, Intermedia’s own website was unavailable, so affected customers could not even go onto the Intermedia website to check for status updates or open support tickets. Needless to say, their PBX was being bombarded by thousands of irate customers as well, so getting someone on the line for an update wasn’t that easy either. Twitter ended up being the best source of updates, first from other customers who tweeted what info they could glean, and then later from Intermedia’s own Twitter account when they managed to get more caught up and started giving out some official updates.

Today I received their formal RFO (Reasons for Outage) letter via email which goes into great details describing why this outage occurred and what steps they are taking to try to prevent a re-occurrence for the same reasons in future. In a nutshell, there was a hardware failure in one of their EMC SAN devices, and this failure occurred in such a way that prevented the device’s own in-built fault tolerance mechanisms from allowing the SAN to effectively remain “up” – that is, they are saying this is one of those failures that should not have happened. These devices are designed precisely NOT to fail under such circumstances, but nonetheless it did fail.

Intermedia’s letter goes on to describe the actions they are taking along with the hardware vendor to guard against this in future. All very good and well. Now on to the little gem in the letter that I found the most surprising, and from which all technologists with “uptime” responsibility for Software as a Service (SAS) systems would do well to learn from.

Here’s the bit that really caught my attention:

“During the event, our ability to communicate status effectively was hindered by an outage of our corporate communication tools until 9:50 a.m. PST. The databases for www.Intermedia.net, Intermedia’s client control panel and Intermedia’s trouble ticket system were located on the affected SAN and therefore were not available during the SAN event. These systems were restored as soon as the SAN performance issue was resolved. All available personnel were directed to answer incoming customer calls. Intermedia logged over 2,000 incoming calls to our PBX and effectively answered more than 1,000 of those calls.”

In hindsight it seems pretty obvious doesn’t it? Why locate your “corporate communications tools” and “trouble ticket system” on the same infrastructure as the core service that you provide? In this case, it might have been the thinking that the EMC SAN just couldn’t possibly fail as it was inherently designed to be fault tolerant, and indeed, EMC SANs are extremely heavy duty devices with very good track records for what they do. Yet fail it did and with it, came down key parts of the foundation, all in one go. Or maybe it was to save on costs. Or maybe it was just a careless oversight. We don’t really know why, but we do know it was implemented that way and that it was clearly a flawed design decision.

Naturally, Intermedia themselves now intend to fix this:

“As a high priority for completion, no later than Q2, Intermedia will also be isolating corporate communication infrastructure from the same infrastructure that provides our Exchange services, guaranteeing that we will be able to communicate effectively with clients at all times during a service interruption. “

Again, this revision might seem like the obvious system and network design that should have been implemented from the get-go, especially for a SAS provider as long in the business and as large as Intermedia. But yet it was not done as such, and it took an outage on this scale to force a change that now seems so obvious in hindsight. Certainly a lesson we can all learn from.

Now one would expect that since you are paying the same amount of money to Amazon for each server instance created of this same type and size, that you should be getting the same performance out of each one. Sadly, that is a very wrong assumption, as our sys admin found out.

It turns out that the underlying hardware for each instance created impacts the actual performance that each instance gives you, even though the instances are all virtualized and marketed by Amazon as if they are all created equal. In our case, we found that the different underlying hardware that the virtual instance sits on has a significant impact on application performance, at least with respect to MySQL database performance. Instances that were created on machines with AMD’s Opteron 270 processors (2ghz 1mb L2 cache) showed significantly poorer MySQL performance compared to instances created on machines with Intel’s Xeon e5430 processors (2.66ghz 6mb L2 cache). Well, the hardware techies among you out there might be saying “well duh… of course the Xeon will spank with the Opteron, tell me something I don’t already know.” But that’s not the point.

The point is in both cases, the EC2 customer is paying the same for an instance that is marketed as having identical compute units (i.e. processing power), but the reality is very different. Bear in mind that one cannot select what underlying hardware you want your instances to be powered up on – what we did was simply keep destroying and creating new instances until we found that the new instance was created on the Xeon-based hardware that we wanted (TIP: from the Linux shell of the new instance, run this to see what hardware your instance was created on: cat /proc/cpuinfo).

Moral: while cloud computing with Amazon (or likely any other vendor of this ilk) has definite pluses, there are hidden gotchas that they don’t tell you about.

So today we get an email from them soliciting feedback to their SimpleDB service, which we recently tried out for a few days, but found somewhat lacking for our needs. The email goes like this:

Dear OleOle,
Amazon Web Services is constantly striving to improve our customers’ experience using our products. We particularly want feedback from our new users.
On 6/23/2009, you signed up for Amazon SimpleDB. Please share your experience about getting started with Amazon Web Services by completing the following survey (9 questions): Amazon SimpleDB Getting Started Survey [This last bit being a link to their survey]. etc etc.

So “Great!”, thinks me, they are being proactive, and hopefully they will improve this service and make it useful for us. Alas, not so fast. The link to their “survey” goes to “Cannot Find Server” – basically that domain for the survey doesn’t resolve in DNS.

Maybe they should have created the sub-domain first?

That’s just so sloppy. Incompetent even. Is Amazon getting too big for it’s britches?

It’s a long article, but very interesting and if you have any interest in keeping a tight reign over the security of data that you keep online (email etc), you owe it yourself to give the TC post a thorough read. Now that we have details as to exactly what occurred and how it was done, my head is spinning with the myriad number of security issues raised by this incident. I plan to write a series of posts in the coming days discussing these issues in greater detail.

I quote here the TechCrunch summary of the attack:

1. HC accessed Gmail for a Twitter employee by using the password recovery feature that sends a reset link to a secondary email. In this case the secondary email was an expired Hotmail account, he simply registered it, clicked the link and reset the password. Gmail was then owned.
2. HC then read emails to guess what the original Gmail password was successfully and reset the password so the Twitter employee would not notice the account had changed.
3. HC then used the same password to access the employee’s Twitter email on Google Apps for your domain, getting access to a gold mine of sensitive company information from emails and, particularly, email attachments.
4. HC then used this information along with additional password guesses and resets to take control of other Twitter employee personal and work emails.
5. HC then used the same username/password combinations and password reset features to access AT&T, MobileMe, Amazon and iTunes, among other services. A security hole in iTunes gave HC access to full credit card information in clear text. HC now also had control of Twitter’s domain names at GoDaddy.
6. Even at this point, Twitter had absolutely no idea they had been compromised.

Source: TechCrunch

WOW! So many things jump right out at me from reading this, including:

1. Sloppy email and password account management by Twitter employees concerned
2. Dangers of mixing work and personal email activities
3. What kind of online footprint you leave by your public participation in social networks, and how vulnerable to attack that can make you (actually this is hinted at not from the above summary account but other details in the TC post)

More to come on this subject in the next few days.

This post is jumping right into the middle of things so to speak where we are already well and fully entrenched in Amazon’s cloud now, and having to deal with the many issues that crop up from time to time that they don’t tell you about in the marketing material.

So on to the subject of today’s post:

We noticed yesterday that all our new EC2 app server instances were booting up in Amazon’s Zone US-east-1d, whereas up till that point in time we had always used Zone US-east-1b with no issues. Several core parts of our system including db servers, load balancers and memcache servers are in Zone US-east 1b.

What’s going on? Well apparently zone 1b is at or near capacity and when we try to force a new instance to be in that zone, we get a message saying “insufficient capacity” , which means any new app server instances that start are not likely to have any chance to be in the same zone as our dbs, load balancers and memcache servers.

Amazon Zone 1b Over Capacity

Furthermore, because our EC2 app server instances all autoscale via Rightscale, they are always short lived – constantly being terminated when capacity is not needed and new ones coming online automatically when load spikes up at certain times of day. The autoscaling is fantastic, a true thing of beauty to behold when you see it action and really makes “utility computing” a reality for us. However, it also means that all of our app server instances are now in zone 1d and not our preferred zone 1b.

And why does this matter? LATENCY is why! We performed some basic testing of ping times between zones:

1b to 1b – average ping time between a server in each zone: 0.45ms
1a or 1d to 1b – average ping time between a server in each zone: 1.9ms

That is a whopping 4 times increase in latency when going across zones versus having all your servers in the same zone. When you have a multitude of calls going on between servers (app to db, app to memcache, and back again, load balancer to app, etc.) in the context of a single web request from a user, that 4 times increase in latency becomes very noticeable, even though we are talking milliseconds differences with each single call.

What can we do about it? We could migrate everything over to zone 1d, but then that seems like a stop gap solution. There’s no guarantee that 1d won’t run out of capacity, in fact, it’s almost certain to at some point, and when that happens, it will force us to have to migrate yet again to keep all our servers in the same zone.

Maybe this is just one of the pitfalls of this type of cloud computing platform, but I can’t get over the feeling that accepting a 4 times increase in latency is just not acceptable.

NOTE: I am well aware that Amazon touts having multiple zones to put your servers in as a plus to avoid a single point of failure as each zone is in a different data centre (and possibly geography). And we have all seen recently what can happen when something takes out a Rackspace data centre (trips on a powercord, network outage, car crashes into a generator, whatever…). But the cost of this is a lot of added latency and it should be a choice that we get to make whether we want to accept that latency or not.