Message Logs Delays

Incident Report for Courier

Resolved

The data stream is unblocked, and the message logs queue is resolved and flowing normally.
Posted Oct 21, 2024 - 11:51 PDT

Monitoring

The fix has landed in production and the team is monitoring the message log queue. Message event logs should be flowing normally.
Posted Oct 21, 2024 - 11:30 PDT

Update

The team encountered an issue with testing the fix and reverted the update. We are publishing a new update that should resolve the backlogged message logs.
Posted Oct 21, 2024 - 10:19 PDT

Identified

The team is testing out a fix to reduce the bottlenecked log lines before releasing to production.
Posted Oct 21, 2024 - 09:07 PDT

Update

We are continuing to investigate this issue.
Posted Oct 21, 2024 - 08:56 PDT

Investigating

The Courier team is investigating an issue with the event logger for message event logs hitting a bottleneck. The team is actively investigating. Messages are still sending.
Posted Oct 21, 2024 - 08:56 PDT
This incident affected: Observability.