Email sending and other asynchronous actions, such as subscriber imports, were delayed from 14:10 to 14:50 UTC on July 30th.
Asynchronous action processing was delayed for about half an hour because our workers were throttled by the CloudFront API while processing tasks. The workers kept retrying, which caused the queue to halt. These actions were still processed during these time, but very slowly.
How did we detect the issue?
We noticed the issue as were conducting infrastructure work and noticed that some test actions were taking too long. Unfortunately, our automated alerting failed, as it depends on this same system to function properly.
How did we mitigate the issue?
The CloudFront API was aggressively throttling us because of some manual infrastructure work we were doing. We stopped the manual operation, which made CloudFront stop throttling us and let the asynchronous action processing queue continue working.
How will we prevent this from happening again?
We will look into why our alerting didn't notify us of asynchronous actions taking too long to be processed, and fix it so an engineer gets paged when this happens.
We will implement stronger tooling in our codebase to make sure that asynchronous actions can't take that long to process in the first place.