Question 1

Can this crawl my site for me?

Accepted Answer

Yes. Give it your homepage and it follows internal links up to a few levels deep and up to 150 pages, skipping redirects, errors, noindex pages and any path prefixes you exclude.

Question 2

Should I block AI crawlers?

Accepted Answer

Blocking training crawlers (GPTBot, CCBot, Google-Extended) doesn't affect whether ChatGPT, Perplexity or Claude can cite you — blocking their search agents does. The tool can write either rule set; it recommends blocking training only unless you want out of AI search entirely.