Our crawler

Kairos Reports runs a small crawler that fetches public news pages and public documents so that every claim we publish can be traced to its source. It identifies itself on every request. It does not log in anywhere, does not fetch anything behind a paywall, and does not train models.

How it identifies itself

Every request carries these two headers. The names still reference The Tayo Report, the engine this masthead grew out of; the identity is the same operator.

User-Agent: The Tayo Report Crawler -- thetayoreport.com/about/crawler
From: crawler@thetayoreport.com

What it fetches

• Article pages and RSS feeds from a fixed list of news outlets, so we can attribute what they reported.
• Public documents: gazettes, court filings, budgets, reports, press statements.
• Nothing else. No comment sections, no user profiles. Publishers' photographs are not reused: story pages and cards carry our own section marks unless a picture is licensed for reuse.

It fetches at a low rate per host and backs off on any error response.

How to block it

The crawler reads robots.txt on every host before it fetches anything there and obeys it for its User-Agent (since 20 October 2026). A disallowed feed is dropped and the refusal is logged. Two more methods also work:

1. Match the User-Agent string above at your server or CDN and return 403. Every request we make carries it, so blocking it blocks everything we fetch.
2. Email crawler@thetayoreport.com with your domain. We add it to the exclusion list, usually within a day.

Being excluded from the crawler does not stop us citing your published work; it stops us fetching it automatically.