WebAssetKit

SEO tools

Robots.txt Generator

A robots.txt file tells search engine crawlers which parts of your site they may visit. Choose what to allow or keep out, add your sitemap address and download the file to upload to your site’s root.

Loading tool…

Adds a block for: . Well-behaved crawlers obey this; others may not.

    Upload the file to the root of your site so it is available at https://your-site/robots.txt.

    How to use the Robots.txt Generator

    1. Choose to allow everything, keep some folders out, or block the site.
    2. If you keep folders out, list one path per line, starting with /.
    3. Add your sitemap address, then copy or download the file.
    4. Upload it so it opens at https://your-site/robots.txt.

    Tips for better results

    • Do not block CSS or JavaScript files that your pages need. Search engines use them to understand the page.
    • Add the Sitemap line even for small sites. It is the simplest way to point crawlers to your pages.
    • Test the file after uploading by opening the address in your browser.

    Limitations you should know about

    • Robots.txt is a request, not security. Polite crawlers follow it, others ignore it, and blocked addresses can still appear in results if other sites link to them. Never use it to hide private data.
    • To keep a page out of search results, use a noindex tag or password protection. A page blocked in robots.txt cannot be crawled, so the engine may never see the noindex tag.
    • The list of AI crawler names changes over time, and blocking them may not stop every company that collects web text.

    Frequently asked questions

    Where does robots.txt go?

    In the top folder of your site, so that it is reachable at your-domain/robots.txt. Each subdomain needs its own file.

    Should I block the whole site?

    Only for sites that should not be found yet, such as a test copy. If you forget to remove it, your site will disappear from search.

    Can robots.txt remove a page from Google?

    Not reliably. Use a noindex tag, password protection, or the removal tool in Google Search Console.

    Guides that use this tool

    Related tools