User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: /wordpress/wp-admin Disallow: /wordpress/wp-includes Disallow: /wordpress/wp-content/plugins Disallow: /wordpress/wp-content/cache Disallow: /wordpress/wp-content/themes Disallow: /wordpress/trackback Disallow: /wordpress/feed Disallow: /wordpress/comments Disallow: /wordpress/category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # http://www.webmasterworld.com/robots.txt has a long list of active robots you might want to block. # Some of these (and many others) ignore robots.txt, and are forcibly blocked in .htaccess. # User-agent: sitecheck.internetseer.com User-agent: MarcoPolo User-agent: Nutch User-agent: Zao User-agent: semanticdiscovery User-agent: PubCrawl User-agent: TurnitinBot User-agent: NPbot User-agent: psbot User-agent: baiduspider User-agent: larbin User-agent: ia_archiver User-agent: NationalDirectory User-agent: LNSpiderguy User-agent: Teleport User-agent: MIIxpc User-agent: asterias User-agent: lwp-trivial User-agent: LinkWalker User-agent: cosmos User-agent: MSIECrawler User-agent: pompos User-agent: Generic User-agent: WebSearchBench User-agent: almaden User-agent: k2spider User-agent: curl User-agent: Wget Disallow: /