Incident Summary
Issue: Temporary service disruption
Following the deployment of a software update, an unexpected database performance issue caused elevated load, resulting in a temporary service outage.
The deployment was immediately rolled back; however, previously initiated API requests continued to place load on the database. As part of the recovery process, GraphQL access was temporarily disabled while long-running database queries were cleared. Once the database returned to normal operation, GraphQL access was restored.
Timeline
- 2:08 PM: Software update deployed.
- Immediately after: Elevated database load resulted in service disruption.
- Recovery actions: Deployment rolled back, GraphQL access temporarily disabled, long-running queries cleared.
- 3:04 PM: Services fully restored.
We have identified the cause of the issue and are reviewing additional safeguards to prevent similar events during future deployments.
Stuart