People now ask AI to research, compare and buy. Agents skip the sites they can’t read. Kennen shows you what to fix, from search basics to agent readiness.
For twenty years, websites were built for people and for Google. People scroll, click around and forgive a slow page. Agents don’t. They fetch your pages, read what’s in them, compare you with others and act, or move on to a business they can work with.
When an agent can’t read your site, it answers anyway, with someone else’s facts. Your analytics never show the visit that didn’t happen.
robots.txt or a firewall turns away the assistant your customer just asked.
Prices and stock only appear after JavaScript runs, and most AI crawlers don’t run it.
Source: Vercel, December 2024Titles, headings and page facts that don’t say what you sell, so an agent can’t match you to what your customer asked.
A form or checkout an agent can’t finish, so it books or buys elsewhere.
Search engines, answer engines and agents read the same pages. Get the basics right and people find you today. Build agent readiness on top and agents choose you tomorrow. Kennen checks all three.
Kennen’s Agent readiness screen for acme.com: the six steps engines and agents go through, in order, with 1 problem under “Are they allowed in?”, robots.txt blocks Claude-User, and 1 ignored.
Checks in six steps, from found to acted on
Are they allowed in?
Problem · Whole site
Anthropic won’t fetch these pages when someone asks their assistant to read them.
From acme.com/robots.txt, line 3
User-agent: ClaudeBot
User-agent: Claude-User
Disallow: /pricing
How to fix
Remove the rules that block Claude-User, or narrow them to the paths you mean to keep out. If the block is intended, choose Ignore.
Fixed when robots.txt lets Claude-User fetch the pages it blocked.
With a CLI and an MCP server
kennen causes list --site acme.com --state open
id title reach
01a130e3-a620-7c41-9d10-5e8a3c2b4f61 robots.txt blocks Claude-User 312
kennen causes brief 01a130e3-a620-7c41-9d10-5e8a3c2b4f61
# Kennen brief: acme.com
Site: https://acme.com/. It is custom code you can change.
From crawl 2, finished 2026-10-19 16:20 UTC. 1 problem, in step order.
## 1. robots.txt blocks Claude-User
Step: Are they allowed in?
Scope: Whole site
What happens: Anthropic won’t fetch these pages when someone asks their assistant to read them.
A new shopping agent, a new bot to allow, a new standard everyone says you need. Most of it doesn’t matter for your site, and some of it matters a lot. Kennen follows the changes, checks your site again when they land, and tells you which ones are worth your time.
Is the agent your customers just started using allowed in, and can it read you?
Agent payments, MCP and what comes next: whether your site needs it yet, and what it takes.
A redesign or a new plugin that quietly locks agents out gets caught.
An agent can find your site, is allowed in, can read and understand what you offer, and can act on it, like booking or buying, without a person stepping in.
No, but it starts there. Agents read the same pages search engines do, so the fundamentals come first. Readiness adds what agents need on top.
Agents are a small share of visits today and growing fast. The fixes are cheap now, and the same ones help your search and AI answers today.
Not to start. Most of what agents need is in your pages. Kennen tells you when an API or an agent standard would make the difference.
