User-Agent: * Allow: / Allow: /api/search Allow: /api/ai-search Allow: /api/repositories Allow: /api/metadata-tags Allow: /api/mcp Allow: /api/health Disallow: /api/ Disallow: /login Disallow: /consent Disallow: /admin/ Disallow: /og Crawl-delay: 5 User-Agent: ChatGPT-User Allow: / User-Agent: OAI-SearchBot Allow: / User-Agent: Claude-Web Allow: / User-Agent: Claude-User Allow: / User-Agent: Claude-SearchBot Allow: / User-Agent: Perplexity-User Allow: / User-Agent: DuckAssistBot Allow: / User-Agent: MistralAI-User Allow: / User-Agent: Meta-ExternalFetcher Allow: / User-Agent: facebookexternalhit Allow: / User-Agent: Googlebot Allow: / User-Agent: GoogleOther Allow: / User-Agent: Bingbot Allow: / User-Agent: BingPreview Allow: / User-Agent: Applebot Allow: / User-Agent: DuckDuckBot Allow: / User-Agent: Slurp Allow: / User-Agent: YandexBot Allow: / User-Agent: LinkedInBot Allow: / User-Agent: Twitterbot Allow: / User-Agent: GPTBot Allow: / Crawl-delay: 10 User-Agent: ClaudeBot Allow: / Crawl-delay: 10 User-Agent: anthropic-ai Allow: / Crawl-delay: 10 User-Agent: PerplexityBot Allow: / Crawl-delay: 10 User-Agent: Google-Extended Allow: / Crawl-delay: 10 User-Agent: Applebot-Extended Allow: / Crawl-delay: 10 User-Agent: Meta-ExternalAgent Allow: / Crawl-delay: 10 User-Agent: YouBot Allow: / Crawl-delay: 10 User-Agent: cohere-ai Allow: / Crawl-delay: 10 User-Agent: cohere-training-data-crawler Allow: / Crawl-delay: 10 User-Agent: CCBot Allow: / Crawl-delay: 10 User-Agent: Bytespider Allow: / Crawl-delay: 10 User-Agent: PetalBot Allow: / Crawl-delay: 10 User-Agent: Diffbot Allow: / Crawl-delay: 10 User-Agent: Amazonbot Allow: / Crawl-delay: 10 User-Agent: AhrefsBot Allow: / Allow: /api/search Allow: /api/ai-search Allow: /api/repositories Allow: /api/metadata-tags Allow: /api/mcp Allow: /api/health Disallow: /api/ Disallow: /login Disallow: /consent Disallow: /admin/ Disallow: /og Disallow: /r/ Crawl-delay: 15 User-Agent: SemrushBot Allow: / Allow: /api/search Allow: /api/ai-search Allow: /api/repositories Allow: /api/metadata-tags Allow: /api/mcp Allow: /api/health Disallow: /api/ Disallow: /login Disallow: /consent Disallow: /admin/ Disallow: /og Disallow: /r/ Crawl-delay: 15 User-Agent: MJ12bot Allow: / Allow: /api/search Allow: /api/ai-search Allow: /api/repositories Allow: /api/metadata-tags Allow: /api/mcp Allow: /api/health Disallow: /api/ Disallow: /login Disallow: /consent Disallow: /admin/ Disallow: /og Disallow: /r/ Crawl-delay: 15 User-Agent: DotBot Allow: / Allow: /api/search Allow: /api/ai-search Allow: /api/repositories Allow: /api/metadata-tags Allow: /api/mcp Allow: /api/health Disallow: /api/ Disallow: /login Disallow: /consent Disallow: /admin/ Disallow: /og Disallow: /r/ Crawl-delay: 15 User-Agent: DataForSeoBot Allow: / Allow: /api/search Allow: /api/ai-search Allow: /api/repositories Allow: /api/metadata-tags Allow: /api/mcp Allow: /api/health Disallow: /api/ Disallow: /login Disallow: /consent Disallow: /admin/ Disallow: /og Disallow: /r/ Crawl-delay: 15 User-Agent: BLEXBot Allow: / Allow: /api/search Allow: /api/ai-search Allow: /api/repositories Allow: /api/metadata-tags Allow: /api/mcp Allow: /api/health Disallow: /api/ Disallow: /login Disallow: /consent Disallow: /admin/ Disallow: /og Disallow: /r/ Crawl-delay: 15 User-Agent: Barkrowler Allow: / Allow: /api/search Allow: /api/ai-search Allow: /api/repositories Allow: /api/metadata-tags Allow: /api/mcp Allow: /api/health Disallow: /api/ Disallow: /login Disallow: /consent Disallow: /admin/ Disallow: /og Disallow: /r/ Crawl-delay: 15 User-Agent: rogerbot Allow: / Allow: /api/search Allow: /api/ai-search Allow: /api/repositories Allow: /api/metadata-tags Allow: /api/mcp Allow: /api/health Disallow: /api/ Disallow: /login Disallow: /consent Disallow: /admin/ Disallow: /og Disallow: /r/ Crawl-delay: 15 Content-Signal: ai-train=no, search=yes, ai-input=yes Sitemap: https://awesome-repositories.com/sitemap.xml # llms.txt: https://awesome-repositories.com/llms.txt