A couple of "these books are definitely on The StoryGraph" searches later and I knew that something was up!
Logging into our search server, we could see our cluster was being reported as 'unhealthy' and no amount of restarting or rebooting would bring it back up. 😩
This was incredibly worrying. Search is needed for so many aspects of the product, beyond a user trying to find a book: Goodreads imports, searching for an author or publisher while adding or editing book entries, finding your friends in the Community section, browsing reading challenges, and more.
So much for the gym! 😅
So what had happened? The hardware that our search system ran on just...died.
To make it clear: the hardware failure happened at a provider of our provider. Two software companies removed from us! That's tech for you!
Even though our provider was eventually able to restore our cluster from a recent backup, Rob and I took this as an opportunity to migrate to (what we hoped would be) a better service!
|