🤖 + 🕸️ One story caught my eye this week was the growth in AI crawlers that are gathering instant results for users of various alternatives to web search. Some coverage uses the “bot” name as an inside baseball style reference to robots.txt.

Techmeme: TollBit: traffic from RAG bots, which scrape information in real time, to 266 websites, half of which are from news outlets, grew 49% from Q4 2024 to Q1 2025 (Nitasha Tiku/Washington Post)

By Nitasha Tiku / Washington Post. View the full context on Techmeme.

As for “bots”, if you look back at coverage of early web crawlers, there are some familiar patterns. 20 years ago, coverage of Google Base was mixed but provided a roadmap for where the increasingly machine readable web was heading.

Official Google Blog: First Base

Insights from Googlers into our products, technology, and the Google culture