Use Nimzoa's free Robots.txt Generator to create a properly structured robots.txt file for your website. Configure search engine crawler rules, allow or disallow selected paths, add XML sitemap locations, and generate a plain-text file without manually writing every directive. Preview, copy, or download your robots.txt file directly in your browser.
BROWSER-BASED SEO TOOL
Create a plain-text crawling guide with separate crawler groups, rules and optional sitemap URLs.
Presets replace the current draft. Robots.txt is public and cannot secure private pages.
Use /paths. Wildcard * is allowed; place $ only at a path's end. Percent-encode Unicode paths.
Changes appear immediately when the entire draft is valid.
robots.txt guides cooperative crawlers; it does not guarantee indexing, hide URLs or protect private content.
This guide is generated locally. Crawler behavior varies; robots.txt cannot enforce access control or guarantee indexing.
Start with an open, blocked or private-path crawler preset.
Add crawler-specific Allow/Disallow patterns and optional sitemap URLs.
Review the validated plain text, then copy or download robots.txt.
Nimzoa's free online Robots.txt Generator is a browser-based SEO tool that helps website owners, developers, digital marketers, and SEO professionals create robots.txt files for managing how cooperating web crawlers access website content. Instead of manually writing crawler directives, you can build a structured robots.txt file using configurable user-agent groups, Allow and Disallow rules, optional sitemap URLs, and a live text preview. A robots.txt file provides crawling instructions to search engine bots and other automated crawlers. It is normally placed in the root directory of a website, where compliant crawlers can retrieve it before accessing other URLs. For example, a website at https://example.com typically publishes its robots.txt file at https://example.com/robots.txt. The Robots.txt Generator supports multiple user-agent groups, allowing you to create different instructions for different crawlers. You can configure general rules using the wildcard user-agent (*) or add crawler-specific groups where different behavior is needed. Within each group, Allow and Disallow directives help define which URL paths cooperating crawlers may request. For more advanced configurations, Nimzoa supports wildcard (*) and end-of-path ($) patterns, making it possible to create rules for selected file types, directories, or URL patterns. You can also add sitemap URLs to help crawlers discover the location of your XML sitemaps. An optional Crawl delay field is available for crawlers that support it, although Googlebot does not recognize Crawl delay in robots.txt. The tool includes starter presets for allowing all crawlers, blocking all crawlers, or excluding common private paths. These presets provide a convenient starting point, but every restrictive rule should be reviewed carefully before deployment because an incorrect robots.txt file can prevent search engines from crawling important website pages. As you configure your rules, the generator displays the resulting plain-text robots.txt content and provides validation guidance. You can review the generated file, copy its contents, or download it as robots.txt for placement on your website. Nimzoa's Robots.txt Generator processes the draft locally in your browser without requiring an account or uploading your configuration for file generation. It is designed to simplify robots.txt creation while keeping the final crawling decisions under your control. Importantly, robots.txt is not a security mechanism and does not guarantee that a URL will be excluded from search results. Use authentication to protect private content, and use appropriate indexing controls when you need to prevent eligible pages from appearing in search engines.
Create a structured robots.txt file without manually writing every crawler directive. Nimzoa supports multiple user-agent groups, Allow and Disallow rules, wildcard and end-of-path patterns, optional sitemap URLs, and crawler-specific Crawl-delay settings. Start with a preset, review the live plain-text preview, and copy or download your file directly in your browser. Built-in validation guidance helps identify common configuration issues before deployment.
Your robots.txt configuration is generated locally in your browser. Nimzoa does not upload or store your draft for file generation. Remember that a published robots.txt file is publicly accessible and cannot protect confidential URLs or guarantee that pages will be excluded from search results.
A robots.txt generator creates a plain-text file containing crawling instructions for cooperating search engine bots and other web crawlers. Nimzoa lets you configure user-agent groups, Allow and Disallow rules, sitemap URLs, and supported path patterns.
No. Robots.txt is publicly accessible guidance for cooperating crawlers, not a security mechanism. Use authentication or appropriate server-side access controls to protect confidential pages and files.
Not necessarily. A URL blocked from crawling may still be discovered through links and appear in search results. To request exclusion from Google Search, use an appropriate noindex directive on content Google can access.
Yes. Robots.txt supports user-agent groups, allowing different crawling instructions for different crawlers. The applicable rules depend on each crawler's matching behavior.
For crawlers like Googlebot that support them, an asterisk (*) matches any sequence of characters, while a dollar sign ($) anchors a pattern to the end of a URL path and query string. Support varies by crawler.
No. Googlebot does not support the Crawl delay directive in robots.txt. Other crawlers may support it, so check their documentation before relying on this setting.
Publish robots.txt at the root of the applicable website host, such as https://example.com/robots.txt. Add a Sitemap directive containing the full URL of your actual XML sitemap, such as https://example.com/sitemap.xml.
Use a configuration appropriate to your WordPress installation, review any existing robots.txt response, and verify your actual sitemap location. Avoid blindly copying restrictive rules that could block important pages or rendering resources.
Google does not support noindex as a robots.txt directive. Use a supported robots meta tag or X-Robots-Tag HTTP header, and ensure Google can crawl the resource to discover the directive.
Nimzoa provides a live robots.txt preview and validation guidance, but it does not simulate Googlebot access for individual URLs. Use appropriate crawler-specific verification tools to check deployed rules.
A robots.txt file is one of the foundational technical files used to communicate crawling preferences to search engines and other automated web crawlers. It helps website owners specify which parts of a website cooperating crawlers are allowed to request and which paths should not be crawled.
A correctly configured robots.txt file can help manage crawler activity and reduce unnecessary requests to selected website areas. However, incorrect rules can prevent search engines from accessing important content, including pages, images, stylesheets, and other resources needed to understand a website.
This guide explains robots.txt syntax, crawler directives, sitemap entries, practical examples, common mistakes, WordPress considerations, and the difference between crawling restrictions and search indexing.
A robots.txt file is a publicly accessible plain-text file containing instructions for web crawlers.
It is part of the Robots Exclusion Protocol, a standard that describes how cooperating crawlers discover and interpret crawling rules.
A typical robots.txt file is located at:
https://example.com/robots.txt
The file contains one or more groups of instructions. Each group identifies a crawler using a User-agent directive and then provides rules such as Allow or Disallow.
A simple robots.txt file might contain:
User-agent: * Disallow: /private/This example asks cooperating crawlers not to request URLs whose paths begin with /private/.
The asterisk in User-agent: * represents crawlers that do not match a more specific applicable user-agent group.
A robots.txt file is not a password-protection system, firewall, or guaranteed indexing control. It provides crawling instructions that compliant crawlers may follow.
Robots.txt can play an important role in technical SEO because search engines need to crawl website resources to discover and understand content.
Website owners may use robots.txt to manage crawling of unnecessary URL patterns, certain duplicate-generating areas, or sections that do not need regular crawler access.
For example, a website with many automatically generated filtering URLs might use carefully designed crawling rules to reduce requests to unimportant combinations.
However, robots.txt does not automatically improve search rankings. Its value depends on whether the rules help crawlers access useful content while avoiding unnecessary crawling.
Blocking important pages or required rendering resources can harm search visibility.
For small websites, a simple robots.txt file may be entirely sufficient. More complicated configurations should only be introduced when there is a clear technical reason.
A robots.txt file uses plain-text directives, generally written as a directive name followed by a colon and a value.
The User-agent directive identifies which crawler or group of crawlers the following rules apply to.
Example:
User-agent: *This represents the general crawler group.
A crawler-specific example is:
User-agent: GooglebotThis identifies rules intended for Googlebot.
Different crawlers may use different names and may interpret supported directives differently. Always check the relevant crawler's documentation.
The Disallow directive identifies URL paths that a cooperating crawler should not request.
Example:
User-agent: * Disallow: /temporary/This asks crawlers not to crawl paths beginning with /temporary/.
A broad rule such as:
Disallow: /requests that crawling be blocked across the entire site for the matching crawler group.
Use this rule carefully. Accidentally deploying it on a public production website can prevent search engines from crawling important content.
The Allow directive can identify paths that are permitted within a broader restriction, for crawlers that support this directive.
Example:
User-agent: * Disallow: /resources/ Allow: /resources/public/For Google-compatible matching behavior, the more specific matching rule takes precedence. Other crawlers may differ in their handling of Allow.
A Sitemap directive provides the absolute URL of an XML sitemap.
Example:
Sitemap: https://example.com/sitemap.xmlA robots.txt file can contain multiple sitemap references.
Sitemap directives help participating crawlers discover sitemap locations, but they do not guarantee that listed pages will be crawled or indexed.
Some crawlers recognize a Crawl-delay directive to control the interval between requests.
Example:
User-agent: ExampleBot Crawl-delay: 10Support for this directive is crawler-dependent.
Googlebot does not support Crawl-delay in robots.txt. Do not rely on it to control Google's crawling behavior.
The following sample robots.txt files demonstrate common configurations.
These examples are educational starting points. Review them against your website structure before publishing.
This permits crawling of the site's paths for cooperating crawlers, subject to any other applicable restrictions.
For many websites, this simple configuration is sufficient.
An empty Disallow directive is another common way to indicate that no paths are disallowed.
This asks cooperating crawlers not to crawl the website.
Warning: Do not use this configuration on a public website that you want search engines to crawl.
It is also not a reliable way to keep private content out of search results or protect it from unauthorized access.
This prevents cooperating crawlers from requesting URLs under the /temporary/ path.
Make sure that the restricted directory does not contain important public pages or resources.
This combines an unrestricted crawling configuration with a sitemap reference.
Replace https://example.com/sitemap.xml with the actual location of your XML sitemap.
This example demonstrates separate user-agent groups.
It is important to understand that Googlebot selects the applicable group according to its matching rules. Do not assume that rules from the general * group will automatically be combined with a more specific Googlebot group.
If you need a restriction to apply to Googlebot, include it in the applicable Googlebot group.
For crawlers that support the pattern syntax, this example matches URL paths ending in .pdf.
This can be useful when selected file types should not be crawled, but it must be reviewed carefully. Blocking a PDF from crawling does not guarantee that its URL cannot appear in search results.
For Google-compatible crawlers, the more specific Allow path can permit crawling under /documents/public/ while the broader /documents/ restriction applies elsewhere.
Test your actual URLs and crawler behavior before deploying complex exceptions.
Some major crawlers, including Googlebot, support special characters in robots.txt path rules.
The asterisk can match a sequence of characters.
Example:
User-agent: * Disallow: /*?sort=For compatible crawlers, this can match URLs containing ?sort= within the path and query portion being evaluated.
Query-parameter restrictions should be used cautiously because they can prevent crawlers from accessing useful page variations.
The dollar sign can match the end of a URL path and query string.
Example:
User-agent: * Disallow: /*.pdf$This targets URLs ending in .pdf, rather than every URL containing that sequence somewhere in its address.
Not every crawler supports the same pattern-matching behavior. Review the documentation of the crawler you intend to control.
One of the most common technical SEO misunderstandings is treating Disallow and noindex as interchangeable.
They serve different purposes.
Feature | Robots.txt Disallow | Noindex directive |
|---|---|---|
Main purpose | Restrict crawling | Request exclusion from search results |
Location | Robots.txt file | HTML meta tag or supported HTTP header |
Must crawler access page to detect it? | No, rule is in robots.txt | Yes, the crawler must retrieve the directive |
Guarantees privacy? | No | No |
Guarantees immediate removal? | No | No |
Not reliably.
Google may still discover a blocked URL through external or internal links. In some circumstances, the URL can appear in search results without its normal page content.
Google does not support noindex as a robots.txt directive.
For an HTML page, a supported noindex implementation might be:
<meta name="robots" content="noindex">For supported non-HTML resources, an appropriate X-Robots-Tag HTTP response header may be used.
The crawler must be able to access the relevant response to discover the directive.
If you block the same URL using robots.txt, Google may be unable to see its noindex instruction.
Use appropriate access controls such as authentication or server-side authorization.
Neither robots.txt nor noindex is a substitute for security.
WordPress websites may use a physical robots.txt file or a dynamically generated robots.txt response, depending on the configuration.
A commonly seen WordPress example is:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap_index.xmlThis example is not a universal recommendation.
Some WordPress installations use different sitemap paths, including /wp-sitemap.xml, and plugins may generate their own sitemap locations.
Before applying a WordPress robots.txt template:
Confirm the actual sitemap URL.
Review whether your theme or plugins require crawlable resources.
Avoid blocking CSS, JavaScript, or images needed for rendering.
Check whether the website already serves a robots.txt response.
Review the behavior of SEO plugins and hosting-level configurations.
A robots.txt generator helps prepare the text, but publishing the file and verifying the live result remain separate steps.
Ecommerce websites often contain product pages, categories, filters, sorting parameters, search results, and customer-related routes.
Some of these URLs may generate large numbers of variations.
A carefully planned robots.txt configuration may help manage crawler access to selected unnecessary URL patterns.
However, broad restrictions can unintentionally block important product or category pages.
Before adding ecommerce crawling rules, consider whether the target URLs need to be crawled for discovery, rendering, canonicalization, or indexing signals.
Do not automatically block every filtered URL or every URL containing a query parameter.
The correct configuration depends on the site's architecture and SEO goals.
A robots.txt generator prepares the file contents. To make the instructions available to crawlers, the file must be published on the relevant website.
A typical workflow is:
Review your website structure and decide which paths should remain crawlable.
Create the necessary crawler groups and rules using the generator.
Add the correct sitemap URL if available.
Review the generated plain-text content.
Download the file as robots.txt.
Publish it at the root of the relevant website host.
Open the published URL to confirm the expected text is served.
Test important URLs and monitor crawl-related reports after deployment.
For example:
https://example.com/robots.txt
Robots.txt rules are scoped to the applicable host, protocol, and port. A file on one subdomain does not automatically control crawling on every other subdomain.
Publishing methods vary depending on whether the website uses WordPress, a custom application, a static hosting platform, or another content management system.
A generated file can be syntactically valid while still containing rules that are inappropriate for your website.
That is why reviewing the actual deployed behavior matters.
Open:
https://yourdomain.com/robots.txt
Confirm that the response contains the expected directives.
For a verified property, Google Search Console provides crawl and indexing information that can help identify robots.txt related issues.
Google also provides documentation and testing guidance for robots.txt behavior.
Check representative URLs, including:
Your homepage
Important category pages
Important product or service pages
Sitemap URLs
Pages under restricted directories
Resources needed to render important pages
Nimzoa's generator provides a live text preview and validation guidance. It does not claim to simulate crawler access for an individual URL, so use appropriate crawler-specific testing methods when that verification is needed.
The rule:
User-agent: * Disallow: /can prevent cooperating crawlers from accessing all site paths.
This is particularly dangerous if a configuration intended for a temporary environment is deployed to production.
Preventing crawlers from accessing required CSS, JavaScript, or other resources can interfere with their understanding of a page.
Robots.txt is publicly accessible, and its contents may reveal the existence of restricted paths.
Protect confidential information with real access controls.
A URL can sometimes remain discoverable or appear in search results even when crawling is disallowed.
Use appropriate indexing controls when the goal is search-result exclusion.
Crawler support varies.
For example, Googlebot does not support Crawl-delay.
A sitemap directive should point to the actual, accessible sitemap URL.
Do not assume every website uses /sitemap.xml.
Specific user-agent groups can change which rules a crawler applies.
Review the complete configuration rather than evaluating each line in isolation.
Robots.txt and XML sitemaps serve complementary purposes.
Robots.txt communicates crawling restrictions and permissions to cooperating crawlers.
An XML sitemap provides a list of URLs that the website owner wants search engines to discover or consider for crawling.
A website may reference its sitemap within robots.txt:
Sitemap: https://example.com/sitemap.xmlHowever, listing a URL in a sitemap does not override a robots.txt crawling restriction.
Similarly, a sitemap does not guarantee that every listed page will be indexed.
Use both mechanisms consistently with your website's technical SEO strategy.
A robots.txt creator can be useful for:
Website owners preparing their first robots.txt file
SEO professionals reviewing crawler directives
Developers configuring website crawling behavior
WordPress administrators managing technical SEO
Ecommerce teams evaluating crawl access
Agencies preparing site migration configurations
Marketing teams checking sitemap references
Beginners can start with a simple configuration, while more experienced users can create multiple crawler groups and advanced path patterns.
The safest approach is to use only the restrictions your website actually needs.
Nimzoa provides a straightforward way to create robots.txt files directly in your browser.
The generator supports multiple user-agent groups, Allow and Disallow directives, wildcard and end-of-path patterns, optional sitemap entries, and crawler-dependent Crawl-delay values.
You can begin with a preset, review the generated plain text, and copy or download the result.
The tool is designed to help reduce manual formatting work while making crawler rules easier to organize.
Before publishing, always review the rules against your website's real structure and verify the live file.
A well-planned robots.txt file can support technical SEO by giving cooperating crawlers clear instructions about which website paths they may request.
Start with a simple configuration, add restrictions only where necessary, and use crawler-specific groups when there is a genuine need.
Use Nimzoa's free Robots.txt Generator above to prepare your file, then publish and verify it on your website.