MCP For Claude Marketplace
Robots.txt Generator

Robots.txt Generator MCP Connector for Claude

A+

Generate syntactically valid robots.txt files by configuring user-agent rules, crawl delays, and sitemap URLs.

3 tools Official Updated Jun 28, 2026 Official Vinkius Partner
More Details Connect to Claude

The Robots.txt Generator is a configuration-driven engine designed to produce precise instructions for web crawlers. By defining specific rules for different user-agents, you can control how bots like Googlebot or Bingbot interact with your site. Use generate_robots_txt to create the final file content, validate_rule_namespace to check for syntax errors in your paths, and get_configuration_summary to audit your current rule complexity. This tool ensures that all directives follow the standard protocol, including proper handling of 'Allow', 'Disallow', 'Crawl-delay', and 'Sitemap' pointers.

robotstxtseoweb-crawlersitemapautomationweb-development

3 tools expose this connector's capabilities to your AI agent.

generate_robots_txt

Generates a robots.txt file based on agent rules

get_configuration_summary

Provides a summary of the current robots.txt configuration

validate_rule_syntax

Validates the syntax of robots.txt rules

See how to talk to your AI agent using Robots.txt Generator.

Generate a robots.txt file for all bots that disallows the '/private/' directory and includes a sitemap at https://mysite.com/sitemap.xml.

User-agent: * Disallow: /private/ Sitemap: https://mysite.com/sitemap.xml

Create rules for Googlebot to allow '/images/' but disallow '/temp/', with a crawl delay of 10 seconds.

User-agent: Googlebot Disallow: /temp/ Allow: /images/ Crawl-delay: 10

Check if my rules for 'BadBot' are valid: agentName: 'BadBot', disallowedPaths: ['/api/v1/'], allowedPaths: []

The syntax for the rules provided is valid.

Use the `generate_robots_txt` tool by providing an array of agent rules containing names, allowed paths, and disallowed paths.

Related Connectors

NEW
CB

CPF/CNPJ Batch Processor MCP

3 tools Official

Validate, format, and clean Brazilian CPF and CNPJ identifiers in bulk.

A+ View details →
NEW
SG

Semester GPA Projector MCP

3 tools Official

Predict your semester GPA and its impact on your cumulative academic standing using performance scenarios.

A+ View details →
NEW
MC

Meat Cooking Timer MCP

3 tools Official

Calculate precise cooking times and target internal temperatures for various meats based on weight, method, and doneness.

A+ View details →
SF

Station F Global Prover MCP

1 tools Official

European startups raise half of what American startups raise — and still think locally. One country. One language. One regulatory framework treated as a burden instead of a barrier that protects them. Station F exists because Europe has 450 million consumers, world-class regulation, and an ambition problem. This tool forces five Station F-level axes: global-first strategy, regulation as moat, capital efficiency, ecosystem density, and continental ambition.

A+ View details →