Elevated API Errors
Incident Report for Heron Data
Postmortem

We apologise for the increased 500 errors and timeouts experienced in the last 10 hours.

We introduced a new index to our database which was partially built and resulted in an “invalid” state. This resulted in long running queries which blocked subsequent reads. We dropped the index, as well as blocking database processes, and are now recovered.

Posted Mar 01, 2023 - 11:29 UTC

Resolved
This incident has been resolved.
Posted Mar 01, 2023 - 10:54 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 01, 2023 - 10:53 UTC
Investigating
We're experiencing an elevated level of API errors and are currently looking into the issue.
Posted Mar 01, 2023 - 09:47 UTC
This incident affected: API (Root, Transactions) and Dashboard.