June 15, 2008

Yahoo Search Engine Spiders Directories From File Paths

I just did an experiment and confirmed that the Yahoo spider will try to search a directory from a file. That is, if it sees a URL like http://example.com/stuff/jump.avi , in addition to retrieving that file, it'll try the URL http://example.com/stuff/ . Though Google won't do that (nor will Microsoft). It's easy to test this yourself if you have a website where you can see server logs. This practice has some significant implications for people who claim that trying truncated URLs is improper behavior and even possibly unauthorized access.

By Seth Finkelstein | posted in infothought | on June 15, 2008 03:50 PM (Infothought permalink)
Seth Finkelstein's Infothought blog (Wikipedia, Google, censorware, and an inside view of net-politics) - Syndicate site (subscribe, RSS)

Subscribe with Bloglines      Subscribe in NewsGator Online  Google Reader or Homepage