Amazon Web Services… Not Quite “Five Nines” Uptime !

…Not five nines,

…or four nines,

…not even three nines (99.9% uptime) !

If you tried to visit some of the Web’s most popular sites for a good part of the day yesterday, July 20, 2008, you were likely disappointed. Sites like WordPress (where this blog is hosted), Twitter, SmugMug and others, were impacted for hours yesterday because they depend on Amazon’s S3 (Simple Storage Service), which went down. Apparently, even some Apple iPhone applications were impacted by the S3 outage. It was the second time in less than six months (the previous outage occured on February 15) that AWS (Amazon Web Services) has experienced a major failure.

Based on what we’ve learned so far about S3, our best guess is that yesterday’s outage was caused by a software bug, a human error of some sort, or as was the case in their February outage, some set of conditions that occured within their system that overwhelmed their ability to handle traffic (interestingly, the latest problem occured early on a Sunday morning… not exactly a time when you would expect a peak load on their system). We view a malicious attack on the service a less likely cause, and hardware or connectivity problems a very unlikely cause. S3 is a decentralized system designed to survive the loss of some of it’s components and still operate normally. In many widespread telecom or network failures suffered by providers and carriers in the past few years, the cause has often been determined to be software related or human error (like a construction crew cutting a fiber optic cable they didn’t know was buried there).

As an aside, here’s some articles about human error that has caused some major outages…

Optus cable culprit found

The Backhoe, The Internet’s Natural Enemy

Cut in Fiber Cable Disrupts Internet Traffic Nationwide

The Backhoe: A Real Cyberthreat

The S3 outages bring to mind another concern among people responsible for the operation of the Internet itself. One of the services that the Internet is built on is DNS (the Domain Name System). The DNS system is what allows your computer to find a website such as this one, from among the millions of computers and websites on the Internet. There is concern among some that even though DNS functionality is spread across many servers on the Internet, in a hierarchical system, that a widespread DNS failure could occur. This would cripple almost all Internet traffic. Worst of all, if there was a major DNS failure, you might not be able to get to this blog ! Heaven forbid.

S3 is a “cloud” storage service. Internet-based computing resources are collectively referred to as cloud computing (see this Businessweek article on cloud computing). In cloud computing, resources that were traditionally located, say, in a company’s data center (disk storage, application software, servers, etc.) are offered by service providers via the Internet. Cloud computing is a relatively new paradigm, and problems similar to what Amazon has experienced are sure to make CIOs and IT managers hesitant to rely on the cloud when they can provide computing resources locally and have greater control over them.

Almost by definition, services offered in the cloud must offer high availability. The uptime standard that is generally used in the telecommunications and computing industries for critical systems is “five nines“, or 99.999% availability. That translates (approximately) to less than five minutes downtime a year, and generally does not include scheduled service outages. In the United States, the public telephone network operated by the Bell System was consistently able to achieve five nines reliability (so Ma Bell wasn’t that bad to us after all, may she rest in peace). Clearly, Amazon’s S3 service has failed this benchmark. It doesn’t even appear that AWS has achieved two nines availability (less than about seven hours downtime per month) this month. That’s utterly dismal performance that is unacceptable for critical systems, and it does not bode well for Amazon’s future in the cloud, or for cloud computing in general.

Interestingly, Amazon’s S3 SLA (Service Level Agreement) states that users are not entitled to a service credit unless their uptime drops below three nines (99.9%) in any month, and even if they fail to achieve two nines (99% uptime) in a month, they will only give users a 25% credit. They must not have a lot of confidence in their ability to provide four nines availability (less than one hour a year of downtime), which Amazon states is one of the design requirements that S3 was built to provide. And if they don’t meet their service levels, will they give their customers a refund? No. It appears all they will offer is a credit to be applied to future service. Not good.

But don’t expect disgruntled S3 customers who have been impacted by Amazon’s Simple Storage System outages to issue press releases critical of Amazon. Paragraph 4.2.4 of their customer agreement specifically prohibits that unless you get their permission first. Incredible.

With an SLA like Amazon’s, and especially because of their outages in the past few months, we might be inclined to use a service such as S3 only to store backup files. We don’t feel that the service is reliable enough to be used to support a live website or other mission critical systems. And even if Amazon had a 100% uptime record, there’s always this to worry about when deciding if you want to depend on services in the cloud (and to think that you were worried about the Y2K problem!).

Perhaps cloud computing is an idea whose time has not yet come.

– Routing By Rumor

The Walmartization of America

<walmart-bashing-mode>

What better way to start my blog than by bashing Wal-mart? After all, they are the retail behemoth that people love to hate. I guess it must be jealousy.

I was never in a Sams Club or Wal-mart store until last year, when a Walmart sprung up about 15 minutes from my home. With their reputation for low wages, poor or non-existent employee benefits and brutal business practices, I wasn’t too eager to become a Wal-mart customer. In fact, the rumor around here is that “Sam’s Club” actually refers to the big stick that Sam Walton would use to beat up his suppliers and the competition. They don’t call us Routing by Rumor for nothing. I did want to check them out, however, to see if their prices were really that good. I consider myself a very savvy shopper. OK, I’ll admit it… I’ve now shopped there several times since they opened.

My first impression was that Wal-mart is K-Mart on steroids. (I hate K-Mart, and won’t shop there.) Wal-mart is very similar to Target stores also. Lots of low-end merchandise. Cheap shoes. Cheap clothes. Not cheap as in inexpensive; cheap as in, well, cheap. I think Target tries to position itself as selling somewhat more upscale clothing. I’ll call it “cheap chic”. And don’t forget that K-Mart has (or had) Martha Stewart. I guess we’re talking higher quality made-in-China merchandise.

Wal-mart does sell many staple items at rock-bottom prices, but many other popular items are priced no lower than other retailers. You can do better on many, perhaps most items at most other chains or supermarkets, especially when an item is on sale, and particularly on grocery items. I felt that some of Wal-mart’s private-label food items I tried were of inferior quality, and not a very good value. Kind of ironic, since one of their private-label grocery brands is called “Great Value”.

Prices seem to jump around a lot at Walmart. Their price “roll backs” come and go, and I’ve seen some items, especially on the last few visits, jump 20%, 25%, or more. I think the bottom line is that you save on one item, but give back what you just saved when you pick up the next item and place it in your basket.

Being the geek that I am, I gravitate to the electronics department of any store I find myself in. I found a few bargains there, but in general, you can do much, much better buying stuff online, a la Amazon or ebay. On some really hot items, like Apple iPods, Wal-mart prices are pretty close to MSRP. I’ve found some small local or regional electronics chains that beat Wal-mart’s prices on electronics by 10% or more.

One thing I’ll concede about Wal-mart, though, is that returns are never a hassle. They seem to be much more consumer-friendly with returns than many large retailers are these days. One other thing that you’ll only find at Wal-mart is that they sell the local newspaper for half-price. I’ve never seen anyone discounting a newspaper at the newsstand, much less selling it for half-price. Obviously, it’s a gimmic, but it is a nice little surprise.

They have gotten a lot of good press lately because of their very low prices for generic prescription drugs. They also have excellent prices on their house-branded OTC drugs. I guess these cheap drugs make up in some small way for the otherwise poor health benefits they offer their employees. Then again, if you can’t afford to see a doctor, you can’t get a prescription for the cheap medicine. Another Wal-mart paradox, I suppose.

What surprises me most about Walmart is how many items are out of stock on each shopping trip. I don’t think it’s so much a case of them having a run on many items, as it is a logistical or management problem. I suppose the really talented, experienced retail people don’t apply for jobs at Walmart. That’s not to say that they don’t have good people working there. They do, and I empathize with anybody that works hard and does not get a decent salary and benefits. Is America better off with the Walmarts, Home Depots and other giants that have decimated virtually all of their retail competition? I think not.

</walmart-bashing-mode>