ebay vs Search Engines

runlevel6 sent us worthwhile article about eBay vs Search Engines. The gist is that a small time auction site was returning search results into eBay. eBay was less than fond of this claiming that people ought to use their search tools. Little company claims this is counterintuitive to the whole point of having a search engine. Check it out.

robots.txt

Of course the correct thing for Ebay to do is put a robots.txt file that limits the area that search engines can traverse. This has been a standard for a long time, and it is understandable to not want search engines to index highly dynamic pages, as the content of those pages would be out of date quickly.

Some things are linkable, some things aren't. Welcome to the dynamic nature of the web.

Ebay kinda justified...for now

I just finished reading an article which seemed to articulate Ebay's point of view a bit better [msnbc.com]

The jist is that the aggregate auction sites continuously spider the 2.8 million items on ebay's site multiple times each day. Given their current scaling problems I can see why ebay would like to go it alone.

Obviously, if they can't solve the scaling problems and the public chooses auction portals over the single ebay site then ebay may have problems. For now they certainly have name recognition to get 'em through the short term.

Andy

I don't get it, eBay...

Let's say I have a grocery store, one of several in a town. It's good, regular customers, I'm doing alright. One day, I open my newspaper to find a new, weekly glossy insert from a firm who goes from grocery store to store, finding the best items for the best price. I guess I could have one of two reactions.

1)I could lavish the guy with praise, because with faith in my service, I know he'll be bringing more customers my way, due in part to name recognition, and in part to the fact that I've got better prices on some items than my competitors.

2)Or, I could have a complete lack of faith in my services, thus basically admitting that others offer a better deal, and try to keep this guy from publishing his rag.

If they don't want this guy searching eBay for deals, how can they even pretend to stand behind their service?

What... AGAIN?

Didn't we see a story on deep linking a few weeks ago? What's wrong with all these companies, thinking that they can take the linking out of HTML?

Once again, as The Onion put it, 79% of Americans 'Just Don't Get It'. The whole point of the web is to make the information "Site Transparent". You don't care where you are and you can drift from one server to another as you wish.

Is there anything we can do to stop this stupidity?

Re:What... AGAIN?

It's one thing to link to other sites, in an entire other thing to try to hijack their realtime database by using a bot to hit the site all day looking for changes.

If bandwith were free I couldn't agree more with the little guy. But Bandwith isn't free and ebay has to pay to let them slow down their services by crawling the whole site all day.
Joseph Elwell.

'Most accurate up to date prices' yeah right.

He also outlined company concerns that outside search results might not display the most up-to-date, accurate information.

This is full of crap. eBay's 'current' bid prices from their own search engine are typically many hours old, I have even seen day-old bid prices displayed on bids.

And for some odd reason, the search seems to miss some items quite often, and I haven't figured out why yet. A search checking for a certain word in the title or description overlooks items that have that word in it. I am sure it isn't a spelling problem, and I believe it isn't a caps problem.

'Deep Linking' is one thing, but constantly spidering a dynamic database is stupid, and eBay has a right to limit it, but I believe that they should keep it open until it really shows signs of causing trouble.

robots.txt anyone?

There's a mechanism for blocking search engines and other spiders. Place robots.txt in the root directory and list dynamic areas which you don't want indexed.

Any search engine which ignores the robot exclusion standard deserves to be spanked in the harshest way possible (DENY ALL FROM *.luser.com comes to mind).

Any web admin who threatens legal action without even implementing this simple procedure (http://www.ebay.com/robots.txt does not exist) deserves to be pointed at, laughed at, and then forced to RTFM until they understand why what they did was wrong.

Sure, I don't think that this would stop someone who was determined to send a spider through their site, but you can't very well blame someone for walking on the lawn if you can't be bothered to put up a 'Keep off the grass' sign.

-D
It's a pity that stupidity isn't more painful.

It's not complicated, ebay...

If you have pages on your site that you only want to be linked from other pages on your site, then you use CGI/mod_perl/apache modules/whatever to check the referer for a valid link source, and you return an error if you didn't get a good referrer.

If people make bots fake their referrer fields to get in anyway, then you've got a case for wire fraud. If not, then you can shut up and be happy.

Listening to companies whine that "Those people are accessing the data we made publically accessible! Make them stop!" is getting annoying.

Re:I don't get it, eBay...

Hmm, seems more like they are complaining that all these guys going around to their store and others are blocking the aisles so their customers can't get through as fast and that some of their customers are getting information that is out of date (sorry, the special on string-beans ended yesterday) and ruining the confidence they have built up with their customers (to further the analogy).

I like the idea posted above. Collect the info and sell it at a fair price on a different server. I don't think eBay should mind people finding info from other sites (assuming they eventually come to eBay and purchase there). If you have some degree of quality control and can throttle the effects of this searching your real-time database at the expense of performance for live users, I don't think there would be a problem.

But what about ad impressions? Does eBay benefot from visitors clicking through multiple pages to search for what they want, seeing an ad each time (I don't know, never been to eBay - just asking).