Robots.txt Generator
Create a robots.txt file with User-agent groups, Allow and Disallow paths, an optional Crawl-delay and your sitemap URL, then copy it or download it.
Group 1
User-agent: * Disallow: /admin/ Disallow: /api/ Sitemap: https://example.com/sitemap.xml
About the Robots.txt Generator
A robots.txt file sits at the root of your site, for example https://example.com/robots.txt, and tells well-behaved crawlers which paths they may fetch. This generator builds one from a form: one or more groups, each naming a crawler and listing paths to disallow or allow, plus an optional Sitemap line that points crawlers to your sitemap.
Each group starts with a User-agent line. Use * for every crawler, or name a specific one: the field suggests common names such as Googlebot, Bingbot, GPTBot and Google-Extended, and you can type any other. A Disallow path blocks every URL that starts with it, so /admin/ also covers /admin/users. Allow is for exceptions inside a blocked path, such as disallowing /private/ but allowing /private/press-kit.pdf. Google and Bing also understand * as a wildcard and $ as the end of the URL, as in /*.pdf$.
The tool keeps the file well-formed. A group with no paths gets an empty Disallow: line, which means nothing is blocked, and warnings appear for paths that don’t start with /, a Sitemap value that isn’t a full URL, and a Crawl-delay inside a Googlebot group, which Google ignores. Bing and some other crawlers do read Crawl-delay.
Remember that robots.txt controls crawling, not indexing or access. A disallowed page can still appear in search results if other sites link to it, and anyone can open it in a browser. To keep a page out of results, let it be crawled and add a noindex robots meta tag with the Meta Tag Generator. Build the sitemap to reference with the XML Sitemap Generator.
How to use the Robots.txt Generator
- 1
Set the User-agent
In Group 1, keep * for all crawlers, or pick or type a crawler name.
- 2
Add Disallow paths
Edit the example paths, click Add Disallow and type a new one, or click a suggested path such as /private/ or /login.
- 3
Add Allow exceptions if needed
Click Add Allow for paths that should stay crawlable inside a disallowed folder.
- 4
Add groups and a sitemap
Use Add Group for crawler-specific rules and enter your sitemap’s full URL. Crawl-delay is optional.
- 5
Copy or download
Check any warnings, click Copy or Download, and upload robots.txt to the root folder of your site.
What it can do
Multiple groups
Separate rules for different crawlers, each with its own Allow and Disallow lists.
Any user-agent
Suggestions for common crawlers, or type the exact name a crawler documents.
Path suggestions
One-click entries for common private paths like /admin/, /api/, /login and /cart.
Well-formed output
Empty groups get an empty Disallow line, blank paths are skipped, and likely mistakes are listed as warnings.
Download as robots.txt
Saves a plain-text file with the right name, ready to upload.
Limitations
- Rules are requests, not security. Badly behaved bots ignore robots.txt, and blocked URLs remain publicly accessible.
- It does not fetch your live robots.txt or test whether a particular URL is blocked by existing rules.
- Crawler names must match what each crawler documents; a misspelled name simply matches nothing.
- The file applies only to the host it is served from: a robots.txt on www.example.com does not cover shop.example.com.
Privacy
The file is assembled in your browser; nothing you enter is sent to FlexyPdf.
Frequently asked questions
Where do I put the robots.txt file?
In the top-level folder of your site, so it loads at https://yourdomain.com/robots.txt. Crawlers don’t look for it in subfolders, and each subdomain needs its own file.
Does Disallow remove a page from Google?
No. It stops Googlebot from fetching the page, but the URL can still be indexed from links elsewhere, usually without a description. To remove a page from results, allow crawling and add a noindex robots meta tag, or protect the page with a password.
How do I block a crawler such as GPTBot or CCBot?
Add a group with that crawler’s user-agent token and Disallow: /, one group per crawler. Tokens such as GPTBot, Google-Extended and CCBot are documented by the companies that run them, but names change and new crawlers appear, so check their current documentation. Obeying robots.txt is voluntary for crawlers.
Should I use Crawl-delay?
Only if a particular crawler is overloading your server and that crawler supports the directive. Bing does; Googlebot ignores it and adjusts its crawl rate based on how your server responds.
What happens if two rules match the same URL?
Google uses the most specific rule, meaning the one with the longest matching path. If an Allow and a Disallow rule are equally specific, Google follows the less restrictive Allow.
Ratings & Reviews
Rate Robots.txt Generator
Help others by sharing your experience. Your rating is shown without your name.
More SEO Tools
View all →Meta Tag Generator
Create title, description, Open Graph and X card meta tags
XML Sitemap Generator
Turn a list of URLs into a sitemap.xml file
SEO Title Analyzer
Check a page title’s length, pixel width and keyword placement
Keyword Density Checker
Count the most repeated words and phrases in your text