Answer: Site owners can block or limit Anthropic's crawler bots by modifying their robots.txt file, using directives such as "User-agent: ClaudeBot" with "Disallow: /" to block a bot from the entire website, or the non-standard Crawl-delay extension to limit crawling activity. Status: Confirmed Entry: tool/claude-apps/privacy-and-data/block-anthropic-crawler Revision: 2 Digest: sha256:1591a5e1238be982 (changes whenever the content changes) Timeline: Created: 2026-11-13T12:04:23Z Last changed: 2026-11-18T12:10:17Z (opt-out requirement: — → modifying the robots.txt file) Last verified: 2026-11-20T12:14:11Z Next check: 2026-11-21T12:14:11Z Check interval: Daily Expected change: No fixed schedule Data: Field | Value | Source | As of block method | Disallow: / | [1] | 2026-11-20 limit method | Crawl-delay: 1 | [1] | 2026-11-20 user-agent example | User-agent: ClaudeBot | [1] | 2026-11-20 opt-out requirement | modifying the robots.txt file | [1] | 2026-11-20 Sources: [1] Anthropic. "Does Anthropic crawl data from the web, and how can site owners block the crawler? | Claude Help Center". https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler Role: primary. Accessed: 2026-11-20T12:14:11Z Quote: "To block a Bot from your entire website, add this to the robots.txt file in your top-level directory. Please do this for every subdomain that you wish to opt out from. An example of this is: User-agent: ClaudeBot Disallow: /" (Supports: block method) Quote: "To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt. An example of this might be: User-agent: ClaudeBot Crawl-delay: 1" (Supports: limit method) Quote: "User-agent: ClaudeBot Crawl-delay: 1" (Supports: user-agent example) Quote: "Opting out of being crawled by Anthropic Bots requires modifying the robots.txt file in the manner above." (Supports: opt-out requirement) Snapshot SHA-256: 18436c38b4f1dda2897e2b39255133161e68e3f0c59a820939816d2e761e4857 (captured 2026-11-18T12:10:18Z) Verification: Method: Automated extraction from [1] with deepseek-flash (JSON output, thinking disabled). Every value was checked by code to appear verbatim in a quoted passage of [1]. On this check the text of [1] was unchanged since 2026-11-18T12:10:17Z, so the values verified then were kept. Cross-check: not-available at 2026-11-20T12:14:11Z. No independent second source is configured for this entry Last successful check: 2026-11-20T12:14:11Z Last attempt: 2026-11-20T12:14:11Z Notes and limitations: Extracted automatically from a single official source (section "Block or limit Anthropic’s bots"). Related: https://latestn.com/tool/claude-apps/privacy-and-data/block-remove-shared-content How do I block or remove content from shared Claude content? https://latestn.com/tool/claude-apps/privacy-and-data/block-websites-web-search How do I block or remove websites from Claude web search results? https://latestn.com/tool/claude-apps/privacy-and-data/bug-bounty-data-use How does Anthropic use data submitted through the Claude Model Safety Bug Bounty Program? https://latestn.com/tool/claude-apps/privacy-and-data/baa-coverage-exclusions What does the Claude BAA cover and which products or features are excluded? https://latestn.com/tool/claude-apps/privacy-and-data/baa-covered-products Which Claude products and features are covered under Anthropic's BAA? https://latestn.com/tool/claude-apps/privacy-and-data/bing-data-processing What information does Claude share with Bing when using visual and interactive content? https://latestn.com/tool/ All entries in AI tools Change log: Time | Event | Change | Source 2026-11-18T12:10:17Z | changed | opt-out requirement: — → modifying the robots.txt file | [1] 2026-11-18T12:10:17Z | changed | user-agent example: — → User-agent: ClaudeBot | [1] 2026-11-18T12:10:17Z | changed | limit method: — → Crawl-delay: 1 | [1] 2026-11-18T12:10:17Z | changed | block method: — → Disallow: / | [1] 2026-11-18T12:10:17Z | changed | alternate methods caveat: blocking IP address(es) → — | [1] 2026-11-18T12:10:17Z | changed | crawl-delay support: Crawl-delay → — | [1] 2026-11-18T12:10:17Z | changed | crawl-delay example: User-agent: ClaudeBot Crawl-delay: 1 → — | [1] 2026-11-18T12:10:17Z | changed | block example: User-agent: ClaudeBot Disallow: / → — | [1] 2026-11-18T12:10:17Z | changed | blocking method: robots.txt → — | [1] 2026-11-13T12:04:23Z | added | Entry created | — Changes for this entry as a feed: https://latestn.com/tool/claude-apps/privacy-and-data/block-anthropic-crawler/latest Challenge this entry: Open challenges: None How: If a value here is wrong or out of date, send what is wrong and a link to evidence. Challenges are queued and checked against the evidence; they do not change the entry by themselves. If the entry changes as a result, the change log says so. Form: https://latestn.com/dispute/tool/claude-apps/privacy-and-data/block-anthropic-crawler POST: https://latestn.com/dispute/tool/claude-apps/privacy-and-data/block-anthropic-crawler with claim, evidence_url, and optionally field and contact (form-encoded or JSON) GET: https://latestn.com/dispute/tool/claude-apps/privacy-and-data/block-anthropic-crawler?claim={what is wrong}&evidence_url={link} How to cite: LatestN. (2026). How can site owners block or limit Anthropic's crawler bots? Revision 2, last verified 2026-11-20T12:14:11Z. https://latestn.com/tool/claude-apps/privacy-and-data/block-anthropic-crawler (accessed 2026-11-21T10:06:14Z). @misc{LatestN_tool_claude_apps_privacy_and_data_block_anthropic_crawler_r2, title = {How can site owners block or limit Anthropic's crawler bots?}, author = {{LatestN}}, year = {2026}, howpublished = {\url{https://latestn.com/tool/claude-apps/privacy-and-data/block-anthropic-crawler}}, note = {Revision 2; digest sha256:1591a5e1238be982; last verified 2026-11-20T12:14:11Z; accessed 2026-11-21T10:06:14Z} } Reuse: License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) Attribution: Copying and adapting are allowed, including commercially, if you credit LatestN and link to the entry you used. Each entry ends with a ready-made citation. URL: https://latestn.com/tool/claude-apps/privacy-and-data/block-anthropic-crawler