Wicked Web is a SpiderTrap.  Specifically it is a system for identifying rogue spiders.  The purpose is for identifying spiders, but not neccessariy stopping them.  In general, we don't mind people getting content from our sites, but we want to know who , specifically when spiders are involved because of their increased impact and effect on our stats ( spiders are removed from traffic reports ).

The jist is this, a link to the cgi is put in the header, footer or other areas of each page.  The link is visually hidden, so people wont follow it, but spiders will, and there fore identify themselves.

Other options are to make the robots.txt file a cgi that registers nice spiders, but just becasue a spider reads robots.txt doesn't mean that it will heed the directives.

Futher, agressive spiders should be stopped in some manner, which might require that bad spiders be fed to firewalls, switches, or to header code for dynamic sites.