IdexAI assistants can only read and cite the pages their crawlers are allowed to reach. Idex reads your store’s robots.txt the way each crawler does, and shows which ones it lets in on Schema & AI files, under AI crawler access.
The 6 crawlers Idex checks
| Crawler | Used by | Why it matters |
|---|---|---|
| OAI-SearchBot | ChatGPT search | ChatGPT search can only show pages it can read. |
| ChatGPT-User | ChatGPT, when a user asks | ChatGPT reads pages this way when a shopper asks about them. |
| GPTBot | OpenAI model training | Used for OpenAI model training. Blocking it doesn't remove you from ChatGPT search. |
| PerplexityBot | Perplexity | Perplexity can only read and cite pages this bot can reach. |
| ClaudeBot | Claude | Anthropic's crawler for Claude. |
| Bingbot | Bing, Copilot | Bing's index feeds Copilot. |
How Idex reads robots.txt
- Idex fetches robots.txt from your store’s main domain.
- Each crawler follows the rules for its own name, or the rules for all crawlers (
*) when there are none for it. - Idex tests a product page address, so the result tells you whether the crawler can read your products.
Each crawler shows as Allowed, Blocked or Couldn’t read. It’s a guide: your store’s firewall can still treat a real crawler differently.
Why GPTBot is only a warning
GPTBot collects pages for training OpenAI’s models. Blocking it doesn’t remove you from ChatGPT search, which uses OAI-SearchBot, so a blocked GPTBot counts as a warning in your score, not a failure.
Unblock a crawler
On Shopify, you change robots.txt by editing the robots.txt.liquid template in your theme. When a crawler other than GPTBot is blocked, Idex shows Show me how, which opens Shopify’s guide to editing robots.txt. Changing the rule takes about two minutes.
Then press Check again. Idex runs a new scan and updates the results.
Password-protected stores
While your store has a password, AI crawlers and Bing can’t read your pages, whatever your robots.txt says.