July 23, 2008

Dear President Bush: Send More Money !

President George W. Bush
The White House
1600 Pennsylvania Avenue NW
Washington, DC 20500

Dear President Bush,

Just wanted to let you know that we received the very generous “economic stimulus” check you sent us recently. It was greatly appreciated.

We thank you.

ExxonMobil thanks you.

The oil speculators thank you.

Saudi Arabia thanks you.

(just to name a few)

Oh, I almost forgot… There’s just one problem. It’s all gone. With each visit to the gas station costing $75.00, your check didn’t go as far as we expected. And now, our gas gauge is on empty again.

Please sir, may we have some more ?

Sincerely,

Routing By Rumor

P.S. - Please also send another check to all of the people who have posted comments below.

July 21, 2008

Amazon Web Services… Not Quite “Five Nines” Uptime !

…Not five nines,

…or four nines,

…not even three nines (99.9% uptime) !

If you tried to visit some of the Web’s most popular sites for a good part of the day yesterday, July 20, 2008, you were likely disappointed. Sites like WordPress (where this blog is hosted), Twitter, SmugMug and others, were impacted for hours yesterday because they depend on Amazon’s S3 (Simple Storage Service), which went down. Apparently, even some Apple iPhone applications were impacted by the S3 outage. It was the second time in less than six months (the previous outage occured on February 15) that AWS (Amazon Web Services) has experienced a major failure.

Based on what we’ve learned so far about S3, our best guess is that yesterday’s outage was caused by a software bug, a human error of some sort, or as was the case in their February outage, some set of conditions that occured within their system that overwhelmed their ability to handle traffic (interestingly, the latest problem occured early on a Sunday morning… not exactly a time when you would expect a peak load on their system). We view a malicious attack on the service a less likely cause, and hardware or connectivity problems a very unlikely cause. S3 is a decentralized system designed to survive the loss of some of it’s components and still operate normally. In many widespread telecom or network failures suffered by providers and carriers in the past few years, the cause has often been determined to be software related or human error (like a construction crew cutting a fiber optic cable they didn’t know was buried there).

As an aside, here’s some articles about human error that has caused some major outages…

Optus cable culprit found

The Backhoe, The Internet’s Natural Enemy

Cut in Fiber Cable Disrupts Internet Traffic Nationwide

The Backhoe: A Real Cyberthreat

The S3 outages bring to mind another concern among people responsible for the operation of the Internet itself. One of the services that the Internet is built on is DNS (the Domain Name System). The DNS system is what allows your computer to find a website such as this one, from among the millions of computers and websites on the Internet. There is concern among some that even though DNS functionality is spread across many servers on the Internet, in a hierarchical system, that a widespread DNS failure could occur. This would cripple almost all Internet traffic. Worst of all, if there was a major DNS failure, you might not be able to get to this blog ! Heaven forbid.

S3 is a “cloud” storage service. Internet-based computing resources are collectively referred to as cloud computing (see this Businessweek article on cloud computing). In cloud computing, resources that were traditionally located, say, in a company’s data center (disk storage, application software, servers, etc.) are offered by service providers via the Internet. Cloud computing is a relatively new paradigm, and problems similar to what Amazon has experienced are sure to make CIOs and IT managers hesitant to rely on the cloud when they can provide computing resources locally and have greater control over them.

Almost by definition, services offered in the cloud must offer high availability. The uptime standard that is generally used in the telecommunications and computing industries for critical systems is “five nines“, or 99.999% availability. That translates (approximately) to less than five minutes downtime a year, and generally does not include scheduled service outages. In the United States, the public telephone network operated by the Bell System was consistently able to achieve five nines reliability (so Ma Bell wasn’t that bad to us after all, may she rest in peace). Clearly, Amazon’s S3 service has failed this benchmark. It doesn’t even appear that AWS has achieved two nines availability (less than about seven hours downtime per month) this month. That’s utterly dismal performance that is unacceptable for critical systems, and it does not bode well for Amazon’s future in the cloud, or for cloud computing in general.

Interestingly, Amazon’s S3 SLA (Service Level Agreement) states that users are not entitled to a service credit unless their uptime drops below three nines (99.9%) in any month, and even if they fail to achieve two nines (99% uptime) in a month, they will only give users a 25% credit. They must not have a lot of confidence in their ability to provide four nines availability (less than one hour a year of downtime), which Amazon states is one of the design requirements that S3 was built to provide. And if they don’t meet their service levels, will they give their customers a refund? No. It appears all they will offer is a credit to be applied to future service. Not good.

But don’t expect disgruntled S3 customers who have been impacted by Amazon’s Simple Storage System outages to issue press releases critical of Amazon. Paragraph 4.2.4 of their customer agreement specifically prohibits that unless you get their permission first. Incredible.

With an SLA like Amazon’s, and especially because of their outages in the past few months, we might be inclined to use a service such as S3 only to store backup files. We don’t feel that the service is reliable enough to be used to support a live website or other mission critical systems. And even if Amazon had a 100% uptime record, there’s always this to worry about when deciding if you want to depend on services in the cloud (and to think that you were worried about the Y2K problem!).

Perhaps cloud computing is an idea whose time has not yet come.

- Routing By Rumor

July 20, 2008

Of Blogs And Bugs

This is one bug we can’t blame on Microsoft.

While writing our last post to this blog, the cute little fellow pictured above came scurrying across the floor, and stopped right in front of us. I’d swear he was looking at our computer screen. Maybe he was thinking of starting his own bug blog.

He sat there long enough for us to scoop him up and sequester him in an empty jar, so we could get our camera and take his mug shot. He’s about an inch and a half or two inches long, and best we can determine, he once had 28 legs, but apparently lost several of them. Think buying shoes for your kids is expensive? Count your blessings that they only have two feet each. It looks like he has antennae fore and aft, although his head is to the right in the above photo.

Given the human need to endear every creature with a name, we have named him “WordPress”. We’ll admit that we are only guessing that he’s a he. If it’s actually Ms. WordPress, our apologies for the gender error. Actually, we’re not even sure if all insects have a gender… Aren’t some both sexes?

Since we don’t have any entomologists on staff here at Routing By Rumor, we are going to enlist the help of our readers. If you know what this cute fellow is, please leave a comment.

All those insect rights people out there will be happy to know that no bugs were harmed in the creation of this article. WordPress was released into the great outdoors after his photo shoot. You should have seen how quickly he moved once we put him on the ground. He was gone in a flash. We suspect, however, that he is going to try and get back into the Routing By Rumor network operations center, so if we spot him again, we’ll post a followup to this story.

- Routing By Rumor

July 17, 2008

When We Speak, Wall Street Listens.

One of the great things about hosting a blog at WordPress.com is the various tools and statistics they provide (thanks, Matt). You can see how much traffic your posts attract, how your visitors are finding your blog, what search terms people are using to find you (but sadly, not which search engine they used), and what hyperlinks in your posts they click on.

We were quite surprised this morning, when we noticed that nasdaq.com was sending traffic our way because of this article we posted yesterday about the U.S. economy. We had mentioned that New York and New Jersey utilities, including Consolidated Edison, had recently gotten steep rate increases approved by regulators, some as much as 25% and 50%.

Within a few hours, visitors to Nasdaq who looked up Consolidated Edison (NYSE: ED), were seeing a link to our blog in the “Comments From The Blogs” section on nasdaq.com. This is yet another example of how blogging has really gained legitimacy, and how mainstream media and even Wall Street have taken notice of what is being discussed by bloggers. To be sure, it is a bit of a trip, realizing that almost anyone with a computer and Internet access can have their voice heard by the world. To us, that’s perhaps the greatest thing about this experiment they call the Internet. It transcends borders (although some regimes try to surpress it), it makes the world a very small place, and it gives you access to views and opinions you would otherwise never hear. How great is that?

Now that Routing By Rumor is (almost) a household name, we want to let the New York Stock Exchange know that we are available, should Dick Grasso’s old job still be open. However, in light of the firestorm that his $140 million retirement package caused, we want to go on record as saying that we will not accept a deferred compensation package of more than $75 million. We think that even Eliot Spitzer, the former New York State Attorney General turned New York Governor (and no friend of Dick Grasso’s), who’s career crashed and burned in a sex scandal, would approve of that.

- Routing By Rumor

July 16, 2008

Need More Proof That The U.S. Economy Is In Trouble? How About Talk Of General Motors Seeking Bankruptcy Protection!

Things are tough in Detroit. For the first time since 1922, General Motors will not pay it’s shareholders a dividend. Even more omnious is speculation that GM may have to seek bankruptcy protection (see NY Times / Associated Press article). There are even calls for dropping GM from the Dow Jones Industrial Average.

While much of what ails GM is related to the sad state of the U.S. economy, not all automobile manufacturers are suffering the way GM currently is. There are many reasons for this, including GM’s reliance on large, gas-guzzling vehicles which have become white elephants, thanks to fuel that is selling between $4 and $5 a gallon.

For us however, GM lost it’s appeal long ago. Not because of the quality of their products, which by and large we’ve been quite satisfied with, but rather because of what we believe is their total lack of respect for their customers. As we have written previously, we are a strong believer in buying products made in the USA, and indeed we’ve owned only US-assembled GM vehicles for the past thirty years.

We’ve never had a good experience with repairs covered under GM’s new vehicle warranties. We have always experienced dealer service that was inept, shoddy, defective, or incomplete, and which in many cases took days or weeks longer than it should have, while our vehicle languished in some dealer’s lot. Almost all warranty repairs we’ve ever taken our vehicles to GM dealerships for have required one or more return visits to the dealer, either because repairs were done incorrectly or not at all. We believe that GM dealerships prefer to not do warranty repairs, and will try to avoid doing them whenever possible. There never seemed to be any incentive to do things correctly. The experiences we’ve had over the years at GM dealerships convinced us to never ever allow them to do non-warranty repairs on our vehicles where we would be paying for the work out-of-pocket.

We’ve dealt with dealership personnel from salespeople to service managers to owners, who have been rude, indifferent and downright obnoxious. Almost as if they were saying “we don’t value you as a customer, and we couldn’t care less about your future business”. GM’s “customer care” has proven to be a worthless farce every time we’ve ever turned to them in an attempt to resolve problems with our vehicles. We decided years ago that our current GM vehicle will be our last one. It’s a bit ironic then, that GM may not be around much longer to sell us, or anyone else, their next vehicle. We think that in large measure, they can blame their current plight on the way they’ve treated their customers. Forget about Harry Gordon Selfridge’s old adage “the customer is always right”. At GM, it has always seemed to us to be more like “the customer is never right”.

We won’t shed a single tear if GM goes belly up tomorrow.

Where was I ? …Oh yeah, the dismal state of the U.S. economy. See what happens when someone mentions GM to me ?

So, GM is hurting big time, banking institutions are in trouble, home foreclosures are at record levels, soup kitchens and food pantries are reporting big increases in families seeking assistance (while at the same time finding it harder to get food donations), and the cost of living is skyrocketing, despite what government inflation figures claim.

IndyMac Bank customers in Burbank, California

Associated Press photo / Kevork Djansezian

The FDIC has compiled a list of 90 banks it says are in danger of failing, victims of the U.S. mortgage crisis.

Consumers are seeing utility bills increase 20, 30, even 50 percent. In New York City, the local electric utility, Consolidated Edison, recently hiked electric rates between 22% and 25%. Because of steep increases in fuel prices, there’s sure to be more double-digit increases in store. In New Jersey, Verizon, a regional U.S. phone company, has gotten permission to raise basic telephone charges 50% over the next three years, and cut the number of free directory assistance calls a subscriber gets in half, while tripling the price. This will no doubt accelerate the trend of consumers dropping traditional phone lines and making their cellphone their only phone. This is not only because of the cost, but also because traditional phone companies like Verizon still don’t know how to be competitive in today’s telecom market where Cable TV, Internet and cellular telecom providers offer highly competitive or all-you-can-eat calling plans that include a smorgasbord of features at no extra charge, and companies like Google are providing FREE directory assistance services. Isn’t Verizon still charging extra every month if you have a Touch-Tone phone? They are clueless.

Healthcare costs are increasing so rapidly that many employers can’t afford to continue providing coverage for their employees, and an increasing number of individuals and families have no health insurance.

Trips to the supermarket induce shock, while a trip to the gas station produces gas pains.

We’re starting to see panicked Americans creating a run on the bank, such as these customers of IndyMac Bank this week.

And, of course, we are still at war.

America is the land of plenty, where we are enjoying a bumper crop of bad news on the economic, political and employment fronts. There seems to be very little good news these days.

- Routing By Rumor