# AI crawler notes for Volgram LLC # Website: https://volgramllc.com # Last Updated: 2026-09-21 # # This file is supplementary. robots.txt is authoritative for access control. # Public marketing content may be used for search, retrieval, citation, and training. # Staff, API intake, and health checks are out of scope. User-agent: * Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Googlebot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Bingbot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: OAI-SearchBot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: ChatGPT-User Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Claude-SearchBot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Claude-User Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: PerplexityBot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Perplexity-User Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Applebot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: GPTBot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: ClaudeBot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Google-Extended Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Applebot-Extended Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: CCBot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Amazonbot Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: meta-externalagent Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Meta-ExternalAgent Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: Bytespider Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ User-agent: cohere-ai Allow: / Disallow: /health Disallow: /staff/ Disallow: /api/ Sitemap: https://volgramllc.com/sitemap.xml Discovery: https://volgramllc.com/llms.txt Discovery: https://volgramllc.com/llms-full.txt Discovery: https://volgramllc.com/llms-small.txt Discovery: https://volgramllc.com/llms.html Discovery: https://volgramllc.com/ai.txt Discovery: https://volgramllc.com/ai.json Discovery: https://volgramllc.com/identity.json Discovery: https://volgramllc.com/brand.txt Discovery: https://volgramllc.com/faq-ai.txt Discovery: https://volgramllc.com/developer-ai.txt Discovery: https://volgramllc.com/robots-ai.txt # Notes: # - Search/retrieval: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, Googlebot, PerplexityBot # - Training: GPTBot, ClaudeBot, Google-Extended, CCBot, and other tokens listed above # - Do not crawl /staff even if a link is guessed # - See /ai.txt for usage permissions