Robots.txt Generator & Validator

Select your website platform (WordPress, Shopify, Next.js, etc.) to start with a preset, or build from scratch. Add Allow/Disallow rules by specifying paths you want to control. Set user-agent targets (like Googlebot), add crawl delays if needed, and include your sitemap URL. Click "Generate" to create your robots.txt file.

"Disallow" blocks search engines from crawling specific paths (e.g., "Disallow: /admin/"). "Allow" explicitly permits crawling, useful for overriding broader Disallow rules. For example, you might disallow "/wp-admin/" but allow "/wp-admin/admin-ajax.php" for functionality.

After generating, use the built-in URL tester to check if specific paths are blocked or allowed. Upload the file to your site root (yoursite.com/robots.txt), then test in Google Search Console's robots.txt Tester. Our generator includes testing instructions for your specific platform.

Use "*" to match any sequence of characters and "$" to match the end of a URL. Examples: "Disallow: /*.pdf$" blocks all PDF files, "Disallow: /*?*" blocks URLs with parameters. Our generator includes wildcard examples and validates your syntax automatically.

Use "*" for all search engines, or target specific bots like "Googlebot", "Bingbot", or "Facebookexternalhit". Most sites only need "*" rules. Specific user-agents are useful when you want different rules for different crawlers (e.g., stricter rules for aggressive bots).

Set crawl-delay only if your server is struggling with bot traffic. Most sites don't need it. If you do, start with 1-5 seconds. Too high delays can slow down indexing. Our generator warns if your delay might hurt SEO and suggests optimal values based on your platform.