KennenBot

KennenBot is the crawler behind Kennen. This page says who runs it, what it does, how hard it works your server, and how to stop it.

Who runs it

Kennen runs KennenBot. If it does something you didn’t expect, write to hello@usekennen.com. Who we are is in the legal notice.

What it does

KennenBot audits a site for someone who added that site to Kennen. It fetches the site’s pages, its robots.txt and its sitemaps, and checks what it finds. It visits no site that nobody added.

User agent and robots token

Every request carries this user agent:

KennenBot/1.0 (+https://usekennen.com/bot)

In a robots.txt file, its token is KennenBot.

How hard it works your server

• It fetches up to 500 pages in one run.
• It sends at most 2 requests at a time to one host, with 250 ms between requests.
• If your robots.txt sets a Crawl-delay, it waits that long between requests, up to 60 seconds.
• If your server answers 429 or 503 with a Retry-After header, it pauses that host for the time given, up to 2 minutes.

Where it crawls from

KennenBot doesn’t crawl sites outside our own tests yet. This page lists its address before it does.

How long we keep what it fetched

Kennen deletes the page content KennenBot fetched after 90 days.

How to stop it

Add these lines to the robots.txt of your site:

User-agent: KennenBot
Disallow: /

KennenBot obeys only groups that name KennenBot, with one exception: it keeps to a Crawl-delayin the group for all bots too. It reads the other rules only to report what they tell other bots; they don’t change what it fetches.

You can also ask us to block your domain. Write to hello@usekennen.com and name the domain. Once it’s on our block list, KennenBot makes no request to it.