🔙 Throwback Thursday for 2025-06-12
AI crawlers are evolving, reshaping how we access instant search results online.
🙏 And a big thanks to all our sponsors! 🙏
(Scroll to the end…)
🤖 + 🕸️ One story caught my eye this week was the growth in AI crawlers that are gathering instant results for users of various alternatives to web search. Some coverage uses the “bot” name as an inside baseball style reference to robots.txt.
Techmeme: TollBit: traffic from RAG bots, which scrape information in real time, to 266 websites, half of which are from news outlets, grew 49% from Q4 2024 to Q1 2025 (Nitasha Tiku/Washington Post)
By Nitasha Tiku / Washington Post. View the full context on Techmeme.
As for “bots”, if you look back at coverage of early web crawlers, there are some familiar patterns. 20 years ago, coverage of Google Base was mixed but provided a roadmap for where the increasingly machine readable web was heading.
Official Google Blog: First Base
Insights from Googlers into our products, technology, and the Google culture
Note: Some images via Unsplash
Disclosure
I am linking to my disclosure.
Cuthrell Consulting LLCAttn: Jay Cuthrell1903 Live Oak St #92Beaufort, NC 28516-0092
To learn about sponsorship opportunities, contact us. 🤓
This section of the newsletter is meant to fund the services stack that enables readership growth and continue free editions to be published each week.
And now, a word from our sponsors
Note: This pseudo-randomized sponsored portion of the newsletter contains referral links that may financially benefit this newsletter.
NexusTek
NexusTek Secure Al Platform ensures organizations can safely unlock the productivity benefits of generative Al across any platform, while ensuring unmatched data security oversight, administration, and observability.
Buttondown
If you're looking for a great newsletter platform, I use and recommend Buttondown. Our readers can get $9 off your first month of Buttondown.
Linode
If you are looking for a reliable virtual private server with simple cloud billing math, I use and recommend Akamai Linode. Our readers can get a $100, 60-day credit on Akamai Linode.
⚠️ This content is not legal or financial advice. ⚠️