Bunbot

The bun.network news crawler. It reads headlines from news sites so readers can see which stories mention the artists, people and places they follow, then sends them to the publisher to read the story.

Operator

bun.network

Purpose

News aggregation: headline, link and publication time

Access

Direct. Runs on its own schedule, never on behalf of a user's browser

Sends readers to

The original article on your site

User agent

Every request identifies itself with this header. The version number changes between releases; match on Bunbot.

Bunbot/0.1.0 (+https://bot.bun.network/)

What it fetches

• RSS and Atom feeds, Google News sitemaps, and a few section front pages.
• An article page only when its feed gives no publication time, to read that one date.
• It stores the headline, the link and the time. It doesn't keep article text or images, and nothing it fetches is used to train generative AI models.

How it behaves

• Reads robots.txt before anything else and follows it. The file is re-read every 24 hours. If it can't be read, the bot stays away and tries again 10 minutes later.
• Honours Crawl-delay (between 1 and 30 seconds), and otherwise waits at least one second between requests to the same host.
• Each feed is polled on its own schedule, between every 3 minutes and once an hour depending on how often it publishes. A few large wire services are polled more often.
• Sends If-None-Match and If-Modified-Since, so an unchanged feed costs you a 304, and accepts gzip.

Blocking it

To keep Bunbot off your whole site, add this to your robots.txt. It takes effect within a day.

User-agent: Bunbot
Disallow: /

Or allow only your feed:

User-agent: Bunbot
Allow: /feed
Disallow: /

Verifying requests

Bunbot publishes its Web Bot Auth keys for HTTP message signatures. A request whose Signature-Agent is "https://bot.bun.network" can be checked against the keys at /.well-known/http-message-signatures-directory; the current key's id is t6h26AC-sop9kEvB2Q7RaY4ujtmzANQEOLtAW5iqhyU.