Recently we had a performance problem that I traced back to shopping site called www.twenga.com. They were generating a lot of requests so I updated the the robots.txt file to slow everyone down and updated the robots.pm file used by awstats to see how much traffic Twenga and a couple others were generating. After a couple of days I decided I had enough and banned Twenga via the robots.txt file. They had consumed over 1.6 GB in the first week of April which was several times more traffic than robots like Google and Bing. Since we do not get any sales from Twenga it was an easy decision.