Success Proofing Your Site

It’s every Web publisher’s dream – that controversial little story posted last night has been picked up by Google News, and traffic to your site has skyrocketed. But before the editor finishes congratulating his intrepid team, the bad news starts to trickle in: Google is sending so much traffic to the site that performance degrades rapidly, and within minutes the site is down completely.

It’s the digital form of “friendly-fire”: a handful of very large sites, like Google News, have the power to send enormous volumes of traffic to Web publishers that are ill-equipped to handle the surge.  

The mechanics of this disruption are identical to those used maliciously in what are known as “denial-of-service” attacks, where criminals harness the collective power of tens of thousands of computers around the world that have been infected with spyware or a virus, and then simply direct all of those computers to request pages from the same target Web site. Unable to distinguish legitimate requests from illegitimate ones, the target site cannot keep up with all of the requests, and eventually becomes unresponsive.

The last few years have seen the rise of a handful of sites that have the power to send extraordinary amounts of traffic to unsuspecting sites within seconds—Google News, Yahoo, Digg and the Drudge Report, to name a few.

Among our customers (some of the largest media sites in the world) we have witnessed traffic on the order of 50,000 to 100,000 requests per hour—the equivalent traffic of a Web site with monthly page views in the 35-72 million range. Most sites with 50 million page views per month have trouble dealing with double the average requests, while sites with capacity in the 1-10 million pages per month range tend to become hopelessly overwhelmed within seconds of receiving this kind of traffic.

Publisher Options

So what is a Web publisher to do? Short of blocking traffic from major sources (a big no-no), there are really two options. First, publishers can build the capacity themselves. This means having around 60 million page views per month of excess capacity available at any given time, which is untenable for any publisher not already publishing hundreds of millions of pages per month.

For the rest of us, a class of solutions has emerged that can help. I think of it as “capacity on demand” and it’s done through service providers than can essentially “rent” you excess capacity when you need it. Akamai is a good example of a business that offers such a service—you pay a higher rate for traffic over your “baseline,” but virtually infinite capacity is available should your site be favored by the likes of Google News.

Over the long term, I expect all publishers to buy bandwidth and serving capabilities the same way we all buy electricity—we simply pay for as much as we need at any given time, and are able to meet big surges when necessary, while at the same time not overpaying for capacity that we don’t need. Not only does it make the most economic sense, but it also helps “success-proof” your site when that next big story gets picked up by the giants.

Dealing With A Traffic Surge

But what happens when your site receives a sudden surge? Clickability hosts three sites for Smithsonian Media: Smithsonian magazine at www.smithsonian.com, Air & Space magazine at www.airspacemag.com and the Smithsonian’s visitor information site, www.gosmithsonian.com. When the controversy over Australian cassowaries hit the Yahoo! homepage in October, the big news at Smithsonian magazine was that having their article and photographs on the colorful birds featured in such a prominent way didn’t bring their site to the ground. Smithsonian.com got through the traffic spike with no ruffled feathers. “We had more page views in eight hours than we had in the entire month of August—our best day of traffic ever,” says Kelly Durkin, senior Web producer at Smithsonian Media. “We didn’t receive a heads up but the site performed just fine.”

John Girard is the founder and CEO of Clickability, a global leader in on demand Web Content Management (WCM). Clickability is headquartered in San Francisco, with offices in New York and London.  John can be reached via email at john@clickability.com.

Post Comment / Discuss This Story - Info/Rules