Localities API Partial Outage

Incident Report for Woosmap

Postmortem

Subject: Service incident on Localities API — 24 June 2026 (resolved)

Summary

On 24 June 2026, between 08:50 and 09:30 UTC (10:50–11:30 CEST), the Localities API experienced about 40 minutes of degraded service. During this window the majority of requests failed (~65%) with HTTP 504 (Gateway Timeout) and successful responses were significantly slower than normal. (p50 ~5s) The service was fully restored by 09:30 UTC. No customer data was lost or exposed.

We're sorry for the disruption and the impact on your integration.

What happened
Demand on the Localities service rose during the morning. Our infrastructure is designed to add capacity automatically as load increases, but on this occasion the automatic scaling did not react quickly enough to keep pace, and the search tier ran short of capacity before additional capacity became available. Queries slowed sharply and exceeded the service's internal time limit, which is why requests were returned as 504 errors. Our engineers restored normal service by manually adding capacity to the search tier, after which response times returned to normal.

How this affected you
During 08:50–09:30 UTC you may have seen:

  • A high rate of HTTP 504 errors on /localities/* requests (peaking around three in four requests).
  • Slower responses on the requests that did succeed.

Outside this window, the service operated normally.

Were you rate-limited?
To relieve load while we restored capacity, we temporarily lowered request rate limits for a small number of high-volume organizations, which would have appeared as HTTP 429 (Too Many Requests) responses for those accounts during the incident. All rate limits have been returned to their normal configured values, no permanent change was made to any account.

What we did

  • Detected the issue via automated monitoring and engaged our operations and Localities teams.
  • Manually added capacity to the search tier to clear the bottleneck and restore service.
  • Restored all temporarily reduced rate limits to normal.

What we're doing to prevent recurrence

  • Improve early alerting to shorten detection and response time. End of Business today.
  • Stop spending capacity on requests that have already timed out, so the system degrades gracefully instead of failing broadly. End of week target for fix.
  • Improve our automatic scaling so it responds earlier to rising demand, rather than after capacity is already strained. Targeted End of Month roll out.
  • Reduce the time it takes for new capacity to become available during a surge, so scaling completes before customers are affected. Targeted End of Month roll out
Posted Jun 30, 2026 - 11:38 CEST

Resolved

This incident has been resolved.
Posted Jun 24, 2026 - 11:37 CEST

Identified

Incorrect Scaling policy.
Posted Jun 24, 2026 - 11:36 CEST

Update

We are continuing to investigate this issue.
Posted Jun 24, 2026 - 11:07 CEST

Investigating

We are currently investigating this issue.
Posted Jun 24, 2026 - 11:07 CEST
This incident affected: Woosmap Localities API.