Tricking Web's 'spiders'


To anyone who's struggled with the Dewey Decimal System in a library, an Internet search engine looks like a gift from heaven. Just type in whatever you're looking for, and seconds later, a list of relevant Web pages appears.

But how relevant are those pages? And why do the sites at the top of the list show up where they are?

Thanks to competition for top search engine rankings and the economics of the Web, you might not always see links to sites with the best information, but sites with the biggest bankroll or craftiest programmers. Increasingly, merchants are paying search engines for top placement or tinkering with the code of their Web pages to fool search engines into putting them atop the lists.

Type "George W. Bush" into the search prompt at AltaVista, HotBot, or iWon and you'll get a million hits.

But high on each list, long before the White House Web site or the Republican National Committee home page, you'll find a link to Political-, which sells "Lub Ya Dubya" bobblehead dolls bearing a likeness of the commander in chief.

In small print above the top listings are the words "Sponsored Links," a euphemism meaning that the Web site operator or broker has paid the search engine to put the site at the top of the list.

The trend has consumer groups concerned, and the Federal Trade Commission has warned search engine operators to label advertising-supported links clearly.

The stakes are big because search engines are so popular, and many users rely on them to find information. Given a specific term or just a keyword, a search engine will sift through an index of hundreds of millions of Web pages to produce a list of links.

Web page designers say they have to break into the top 30 sites on a search engine list if they want to draw visitors. With surfers ignoring banner ads and newer pop-up windows, search engines are making the most of their position by charging Web sites for top placement when a user enters specific terms.

Tom Steinhagen, who operates Political Pizazz with his wife, Valerie, contracted for the "George W. Bush" placement with Overture Services Inc., which acts as a middleman between advertisers and search engine operators.

Steinhagen pays 7 cents every time someone clicks on one of his sponsored links. He originally listed his company under keywords such as "bobbing head doll." But when stiff competition in the trendy bobble-head market drove the price up to 25 cents per click, Steinhagen shifted his keyword listing to the president's name, which was cheaper.

"We thought it was more cost-effective," he said.

When deciding on the order in which it displays sponsored links, a search engine might take into account the actual relevance of the site. But driving force is still money.

"The more that you've paid, the more likely you are to rank higher," said Danny Sullivan, editor of, an online industry publication. During peak shopping seasons, the cost for some gift-oriented listings can soar to over a dollar per click.

Steinhagen said shifting to the "George W. Bush" keyword has increased the number of visitors, but not his overall sales. So he's pondering yet another change. The key, he said, is picking the right keywords and search engine optimization service.

Otherwise, he said, "It's kind of like advertising when everybody's television is off."

Until recently, "sponsored listings" appeared under more ambiguous names such as "featured listings" or "recommended sites." Last year, the advertising watchdog group Commercial Alert filed a complaint with the Federal Trade Commission.

"This is a bread-and-butter issue for us," said Gary Ruskin, Commercial Alert's executive director. "We're trying to defend the advertising-editorial line."

Commercial Alert said offending search engines that had gained credibility by ranking sites on content were now posting paid links in exactly the same format with no indication they were ads.

The FTC took no formal action but did send letters to the offenders, warning them to clarify which hits were ads or face possible sanctions.

While a few search engines, including popular, still list paid results brokered through Overture with no special markings, most sites heeded the FTC's warning and indicate which are "sponsored."

Ruskin considers the change a victory: "'Sponsored' means advertising. Everybody understands that."

Outright advertising isn't the only way to climb in the rankings. Many Web site programmers expend considerable effort to attract or even fool search engine "spiders." These are programs deployed by Google and other "crawler" search engines that prowl the Web, looking for new sites and hunting for keywords that form the basis of the search engine's index.

A relevant Web site will generally attract a spider, but some operators employ programming tricks rather than substance. For example, if you run across a hit on a pornographic Web site while searching for tax information, you've probably found a webmaster adept at spider deception.

A common trick is to "display" loads of irrelevant keywords in white type on a white background. The spider sees them, but the viewer doesn't.

A second is embedding misleading terms in a Web page's "metatags," text that doesn't appear on the page but is designed to help search engine indexes. To see these, click on the View menu of your Web browser and choose "Source." This displays the raw text and programming commands that create a Web page.

Yet another tactic is known as cloaking - writing a page for a search engine spider that automatically opens a completely unrelated page when a user clicks on the link.

"It's always likened to an arms race," Sullivan said of the search engines' efforts to thwart these practices.

Porn sites are notorious for these tricks, but at least one Christian-themed Web site has turned the tables by adding such words as "nude," "celebs" and "porn" to its metatags. Sinners beware.

"That's an old trick," said Meghan Morris, who operates Baltimore-based Dragonfly Design. As part of her Web design work, Morris helps clients with search engine "optimization," as the effort to please the spiders is known.

But she says she does it the old-fashioned way - using appropriate keywords in the right places. Her company usually ranks No. 1 on Google under a search for "Baltimore web design."

Unfortunately, "Not every webmaster is honest," said Google software engineer Matt Cutts. Google, generally regarded as the Web's premier search engine, employs a handful of software experts like Cutts to catch and deter such tricksters.

"People who try to do this get their pages taken out very quickly," he said.

Google's software, PageRank, judges sites largely by using "inbound" links." Basically, it ranks Web pages based on the number and quality of external sites that link to it.

"Overall, I would say that Google's goal is to make it so hard to deceive a search engine that it's easier to write a good site in the first place," said Cutts.

Brett Tabke of SearchEngineWatch said inbound linking has proved effective and made it "the most difficult [technique] to optimize for."

"Tricks really aren't working that well anymore. People are realizing they need content," he said. "Content is still what brings people to the site. It's what keeps them at the site and it's what brings them back."

Copyright © 2019, The Baltimore Sun, a Baltimore Sun Media Group publication | Place an Ad