You can write the clearest answer on the internet, but if an AI crawler is blocked, or lands on a page that's empty until JavaScript runs, that answer won't be cited. Technical SEO is the part of AI search most businesses never check — and the easiest place to lose.

Here's the checklist we work through first.

1. Know which crawlers matter

Different AI products use different crawlers, and many companies now separate training crawlers from search crawlers. That distinction matters: you can decline to have content used for model training while still allowing it to appear (and be cited) in AI search answers.

Crawler (user agent)CompanyUsed for
GooglebotGoogleGoogle Search, including AI Overviews
Google-ExtendedGoogleA control token for Gemini model use — Google says it doesn't affect Search
BingbotMicrosoftBing search, which also powers Copilot answers
OAI-SearchBotOpenAIShowing sites in ChatGPT search results
GPTBotOpenAICollecting content for model training
PerplexityBotPerplexityIndexing sites for Perplexity answers
ClaudeBotAnthropicCollecting content for model training

User agents change. Check each company's crawler documentation before you make decisions, and review your settings a few times a year.

2. Set robots.txt on purpose

Many sites block AI crawlers by accident — a security plugin, a CDN "block bots" toggle, or an old blanket rule. Open yourdomain.com/robots.txt and make sure it reflects a decision you actually made.

# Allow search and answer crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Optional: opt out of model training only
User-agent: GPTBot
Disallow: /

Sitemap: https://www.yourdomain.com/sitemap.xml

Key takeaway: Check your CDN and firewall too. Some bot-protection settings block AI crawlers before robots.txt is ever read — your logs will show it as a wall of 403 responses.

3. Don't hide your content behind JavaScript

Google renders JavaScript, but many other crawlers — including several AI crawlers — fetch the raw HTML and move on. If your page is an empty shell until a script runs, that's all they see.

A client-side rendered page shows an empty root element to crawlers; a server-rendered page shows the full content
If the words aren't in the HTML response, a crawler that doesn't run JavaScript can't read or cite them.

Do this: view the page source (not the inspector) of your key pages. If your headings, copy and prices aren't there, use server-side rendering or static generation for those pages.

4. Get indexed — on more than Google

AI answers draw from search indexes. Verify your site in Google Search Console and Bing Webmaster Tools, submit an XML sitemap to both, and fix pages marked "crawled — currently not indexed". Bing also supports IndexNow, which lets you notify it the moment a page changes.

5. Add structured data that clarifies who you are

Structured data won't force a citation, but it removes ambiguity about your business and your content.

  • Organization — name, logo, website, social profiles.
  • LocalBusiness — address, hours, service area (for local businesses).
  • Service or Product — what you sell, with accurate details.
  • Article — author, date published and date modified on blog posts.
  • BreadcrumbList — how pages fit together.

Validate with Google's Rich Results Test and fix warnings on your most important templates first.

6. Keep pages fast and stable

Slow, heavy pages get crawled less often and frustrate the people who do click through from an AI answer. Aim for good Core Web Vitals on mobile, compress images, and avoid layouts that jump around while loading.

7. Clean up duplicates and dead ends

Duplicate pages split signals; broken links and redirect chains waste crawl budget. Use canonical tags, fix internal links that point at redirects, and make sure every important page is linked from somewhere a crawler will find it.

Your quick-scan checklist

  1. Robots.txt reflects deliberate choices for each search and AI crawler.
  2. CDN / firewall isn't blocking legitimate crawlers.
  3. Key content is present in the raw HTML.
  4. Site verified and sitemaps submitted in Google and Bing.
  5. Organization / LocalBusiness and Article structured data in place.
  6. Core Web Vitals in the "good" range on mobile.
  7. No duplicate, orphaned or redirect-chained key pages.