On July 24th, from 15:13 to 21:29 UTC, most links using custom tracking domains were unavailable. This was caused by a badly planned migration of an internal service (webhook-consumer) that these domains depended on.
How did we detect the issue?
Unfortunately, we found out about this issue from user reports, which is unacceptable. While we have added monitoring for these domains precisely to monitor for these kinds of issues, this monitoring was misconfigured and didn't alert us when it started going off.
How did we mitigate the issue?
At 21:12 UTC a support team member paged an engineer after receiving user reports that tracking links using custom domains were broken. After investigating, at 21:30 we reverted the DNS target of the webhook-consumer service back to its Heroku counterpart, which restored service instantly (after DNS cache was busted.)
How will we prevent this from happening again?
Review all our existing monitoring to make sure it's alerting us loudly when something that definitely should work is failing.
When planning future migration work, properly map all dependencies to avoid missing stuff to consider.