Crawling settings

One add-on that decides whether search engines, AI crawlers, and everything else may crawl your archive, separate from who can read it.

Justin Duke

September 12, 2026

Some folks want their archives available so they can link to them, but don't want them to come up in Google results.

Some folks want to prohibit LLMs and other scrapers from crawling their archives.

Both of these are eminently reasonable! We tried to handle a lot of this with a very engineer-brained approach: give you a big ol' textarea where you can type user agent strings to block.

People do not know or care about user agent strings, we realized. So we replaced all of it with a Crawling add-on in Settings → Archives, holding one toggle per kind of crawler.

demo.buttondown.com/settings/archives

This is a live demo. You can view this page on our live demo site, too.

We looked at usage (and our support volume) and found the two biggest reasons/avenues that y'all care about, plus a catch-all for everyone else:

Search engines. Turn it off and every subscribe and archive page tells Google, Bing, and friends not to index it, and your sitemap goes dark. Links you share still unfurl nicely on social media, because we keep the OpenGraph tags around.
AI crawlers. Turn it off and we block a curated list of AI training and assistant crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and a few dozen more) in your robots.txt, tag every page for them, and refuse any subscription attempts they make.
Everything else. Every other crawler: scrapers, SEO tools, archivers, and the long tail of bots that don't say who they are. Turn it off and your robots.txt tells all of them to go away, with an explicit carve-out for the search engines and AI crawlers you've left on.

One honest caveat: robots.txt lives at the root of a domain, so on buttondown.com/your-newsletter we can't write per-newsletter rules into it. The AI crawlers toggle is fully enforced on a custom hosting domain, and advisory everywhere else. Search engine blocking works the same on both, since it rides on per-page tags rather than robots.txt.

The docs have the full rundown of what each toggle does under the hood.

Frequently asked questions

What actually changed for me?

For almost everyone, nothing. All three toggles default to on, which matches how your archive already behaved: search engines could index it and AI crawlers could read it. The only newsletters whose behavior changed are the few dozen that had set one of the old hidden noindex flags or typed entries into the blocked user agents box; those were migrated to the matching toggle so they keep doing what they did before.

What happened to blocked user agents?

If you had entries in the old blocked user agents textarea, we migrated you. A list that named AI crawlers becomes "AI crawlers: off". A list that named Googlebot or Bingbot becomes "Search engines: off". A list that named anything else — vulnerability scanners, SEO bots, a stray archiver — becomes "Everything else: off", which disallows every crawler outside the two named buckets. If you were relying on very fine-grained per-agent rules, the firewall is a better home for them than robots.txt.

Is this paywalled?

No. The whole add-on is available on every plan, including the free one. The only wrinkle is that fully enforcing the AI crawlers and everything else toggles through robots.txt needs a custom hosting domain, which is a paid feature; without one, those toggles still block sign-ups from AI crawlers and tag every response, but can't write to robots.txt.

Buttondown is the last email platform you’ll switch to.

Join your last newsletter platform