1. Crawler access
Start here, because it's binary. Check your robots.txt for these user agents and make sure the ones you want are allowed:
| User agent | Operator | Purpose |
GPTBot | OpenAI | Training and index crawling |
OAI-SearchBot | OpenAI | ChatGPT search index |
ChatGPT-User | OpenAI | Live fetch when a user's question needs it |
ClaudeBot / Claude-User | Anthropic | Index and user-triggered fetch |
PerplexityBot | Perplexity | Index crawling |
Google-Extended | Google | Gemini and AI grounding |
meta-externalagent | Meta | Meta AI |
Two traps. First, blanket AI opt-outs added during the 2023–24 scraping backlash are still live on many sites — check before assuming. Second, robots.txt is only read from the domain root. If your site lives on a subpath (a GitHub Pages project site, for instance), your robots.txt there is decorative.
Verify, don't assume
Check server logs for actual visits from these agents. A permissive robots.txt with a firewall or bot-protection rule silently blocking the same crawlers is a common and invisible failure.
2. Rendering and markup
Not every AI crawler executes JavaScript, and those that do may not wait for slow client-side hydration. Content that only exists after a framework boots is content at risk.
- ✓ Serve meaningful content in the initial HTML response — static generation or server-side rendering.
- ✓ Test with JavaScript disabled: what remains is roughly your worst-case machine view.
- ✓ Use real semantic elements — one
<h1>, ordered headings, real lists and tables.
- ✓ Put important content in text, not baked into images.
3. Structured data that states your identity
This is where GEO diverges most from classic SEO. You are not adding schema to win a rich snippet; you are removing ambiguity about what entity this site represents. A connected graph beats scattered fragments:
- Organization (or LocalBusiness / ProfessionalService) — name, URL, logo, address, contact, area served.
- Person — the founder or key people, linked via
founder and worksFor.
- WebSite and WebPage — tying pages to the publisher.
- Service — what you actually sell, with a provider reference.
- FAQPage — question/answer pairs matching visible content verbatim.
- BreadcrumbList and Article on content pages.
Two rules that matter more than completeness. Use @id to link nodes so the graph is explicit rather than a pile of unrelated objects. And never let schema contradict visible content — mismatched FAQ markup in particular is a known source of manual penalties, and it undermines exactly the trust you're trying to build.
The sameAs signal
Your sameAs array — LinkedIn, GitHub, directory profiles, a Wikipedia entry if you have one — is how you tell an engine which external records refer to you. This is one of the highest-leverage and most-skipped fields.
4. Freshness signals
Recency is weighted heavily. Make it legible:
- ✓ Visible "last updated" dates on content pages.
- ✓ Accurate
dateModified in Article schema.
- ✓ A sitemap with honest
lastmod values.
- ✓ A real refresh cadence — updating substance, not just bumping dates. Cosmetic date changes are detectable and corrosive to trust.
5. llms.txt — useful, oversold
An llms.txt file at your domain root offers a plain-markdown summary of your site for language models. It's cheap to add, it's a clean way to state your canonical description, and we ship it as standard.
But be clear-eyed: adoption by major engines is not established, and it is not a ranking mechanism. Any agency presenting llms.txt as the core of a GEO offering is selling the easiest deliverable rather than the effective one. Crawlability, entity clarity, extractable content and off-site authority do the actual work.
The 30-minute audit
- Fetch
yourdomain.com/robots.txt — are AI agents allowed, and is it at the true root?
- View source with JS disabled — is the content there?
- Run the page through a schema validator — does a connected entity graph appear?
- Check your canonical, og:image and og:url actually match the live URL.
- Confirm sitemap.xml resolves and is submitted in Search Console and Bing Webmaster Tools.
- Search your own brand name and check what the top external sources say about you.
If you'd rather have this scored for you, our free AI visibility checker walks through the same ground in a few minutes.
Frequently asked
Does robots.txt work on GitHub Pages project sites? +
Only partially. Crawlers read robots.txt from the domain root, so on username.github.io/project/ a robots.txt inside the project folder isn't authoritative — the one at the domain root governs. It becomes fully effective once you point a custom domain at the site.
Will adding schema get me cited? +
Not on its own. Schema makes you eligible and unambiguous; it doesn't make you authoritative. Think of it as removing a barrier rather than adding a boost — necessary, not sufficient.
Do I need llms.txt? +
It's cheap and harmless, so add it — but don't expect much from it alone. Prioritize crawl access, rendering, entity schema and off-site corroboration first.
How do I check if AI crawlers are actually visiting? +
Look in your server or CDN access logs for the user agents listed above. If your host doesn't expose logs, Cloudflare and similar CDNs report AI crawler traffic in their analytics.
Keep reading
Want this done for your brand?
Book a free GEO audit: where you appear across 6+ AI engines today, the prompts competitors own, and a prioritized 90-day plan you keep either way.
Book a free audit →