RFO: February 2, 2026 — Service Interruption

Executive Summary

On February 2, 2026, a deployment introduced a configuration change that referenced a file not present in our production build artifacts, causing backend services to become unavailable. API endpoints, automations, and tracking functionality were impacted for 2 hours and 17 minutes in the US region and 2 hours and 42 minutes in the Ireland region. Recovery was prolonged by a concurrent outage at GitHub Actions, our CI/CD provider, which prevented our standard automated rollback. Our team identified the root cause, executed a manual rollback independent of the affected provider, and restored full service across all regions. We have defined targeted action items to prevent recurrence.

Incident Overview

• Affected services: API endpoints (including message sending), automations, webhooks, tracking links, and authentication
• Impact: Requests to backend services returned errors for the duration of the incident. Users were unable to send messages, trigger automations, or access tracking data. No data was lost — requests were rejected before ingestion, so no messages were partially processed or left in an inconsistent state.
• Detection: Our monitoring systems flagged elevated error rates within minutes of the issue beginning.
• Contributing factor: GitHub Actions, our CI/CD provider, experienced a complete outage from 10:35 to 16:30 PST. This overlapped with our incident window and prevented our standard automated rollback from executing, extending the time to resolution.

Timeline of Events

All times in PST.

Root Cause Analysis

The disruption was traced to a configuration change included in the latest release. The change introduced a startup dependency on a utility file that was intended to be bundled with the deployment package. However, the file was not included in the production build artifacts. When backend services attempted to initialize, they were unable to locate the required file and could not start, resulting in all incoming requests being rejected.

This discrepancy was not caught prior to production deployment because the file was present and functioning correctly in the development environment. The difference in how build artifacts are assembled between development and production environments meant the issue only manifested in production.

Mitigation and Resolution

1. Upon identifying the root cause, the team immediately initiated a rollback to the prior known-good release through our standard CI/CD pipeline.
2. A complete outage at GitHub Actions, our CI/CD provider, prevented the automated rollback from completing. The team identified this external dependency and pivoted to an alternative approach.
3. The team executed a manual rollback by retrieving prior deployment artifacts from our backup storage and deploying them directly, bypassing the affected CI/CD pipeline entirely.
4. Services were restored region by region, with the US region confirmed operational at 14:01 PST and the Ireland region at 14:26 PST.
5. Extended monitoring was conducted across all services before declaring the incident fully resolved at 15:13 PST after we were satisfied everything had been stable for over 45 minutes. 

Action Items