Delay in Message Processing

Incident Report for Courier

Resolved

The general pipeline has recovered.
Posted Aug 19, 2024 - 14:35 PDT

Update

Fix has been deployed, and enqueued messages have started to go through slowly. Once the bottleneck clears, messages should start to flow normally.
Posted Aug 19, 2024 - 13:02 PDT

Update

Release is published and building to production. ETA ~45 minutes.
Posted Aug 19, 2024 - 11:56 PDT

Update

The release is live, and the team is monitoring it.
Posted Aug 19, 2024 - 11:50 PDT

Monitoring

Our team has released a revert to address the regression and it's in the process of merging.
Posted Aug 19, 2024 - 11:48 PDT

Identified

The Courier team identified an issue in our health monitoring involving our message event processing. The issue has been identified and a revert is in place.
Posted Aug 19, 2024 - 11:32 PDT
This incident affected: Web Application, API, and Observability.