User-agent: Slurp Crawl-delay: 2 User-agent: Mediapartners-Google Disallow: User-agent: wget Disallow: / User-agent: * Allow: /