robots.txt (1279B)
1 User-agent: AI2Bot 2 User-agent: Ai2Bot-Dolma 3 User-agent: Amazonbot 4 User-agent: anthropic-ai 5 User-agent: Applebot 6 User-agent: Applebot-Extended 7 User-agent: Brightbot 1.0 8 User-agent: Bytespider 9 User-agent: CCBot 10 User-agent: ChatGPT-User 11 User-agent: Claude-Web 12 User-agent: ClaudeBot 13 User-agent: cohere-ai 14 User-agent: cohere-training-data-crawler 15 User-agent: Crawlspace 16 User-agent: Diffbot 17 User-agent: DuckAssistBot 18 User-agent: FacebookBot 19 User-agent: FriendlyCrawler 20 User-agent: Google-Extended 21 User-agent: GoogleOther 22 User-agent: GoogleOther-Image 23 User-agent: GoogleOther-Video 24 User-agent: GPTBot 25 User-agent: iaskspider/2.0 26 User-agent: ICC-Crawler 27 User-agent: ImagesiftBot 28 User-agent: img2dataset 29 User-agent: ISSCyberRiskCrawler 30 User-agent: Kangaroo Bot 31 User-agent: Meta-ExternalAgent 32 User-agent: Meta-ExternalFetcher 33 User-agent: OAI-SearchBot 34 User-agent: omgili 35 User-agent: omgilibot 36 User-agent: PanguBot 37 User-agent: PerplexityBot 38 User-agent: Perplexity‑User 39 User-agent: PetalBot 40 User-agent: Scrapy 41 User-agent: SemrushBot-OCOB 42 User-agent: SemrushBot-SWA 43 User-agent: Sidetrade indexer bot 44 User-agent: Timpibot 45 User-agent: VelenPublicWebCrawler 46 User-agent: Webzio-Extended 47 User-agent: YouBot 48 Disallow: / 49 50 User-agent: Googlebot 51 User-agent: Bingbot 52 Allow: / 53 54 User-agent: * 55 Allow: / 56