Post-Mortem: Service Performance Issue
Date: 07/01/2025
Duration: 40min
Impact: Elevated response times on Distance service and possible 5xx / 4xx errors

Summary
During a routine security update on the service, a disruption occurred due to an issue with a third-party dependency. This eventually led to a server failure. Automatic traffic redistribution has been triggered to the remaining servers. This resulted in high-volume requests on those servers, leading to degraded response times for all users for approximately 30 minutes.

Root Cause
Unforeseen issue with a third-party dependency during routine security updates. Pre-update testing did not account for this specific issue.

Resolution
-Additional server deployed to replace the lost one.
-Rolled back the update to restore service.

Preventative Actions -

-Strengthen testing processes to better cover third-party dependencies.
-Scale up the default configuration to reduce impact of unexpected high volume of requests.

Lessons Learned
Third-party dependencies require closer monitoring and contingency planning to minimize risks during service updates.