Elevated API Errors

Incident Report for Heron Data

Postmortem

We apologise for the increased 500 errors and timeouts experienced in the last 10 hours.

We introduced a new index to our database which was partially built and resulted in an “invalid” state. This resulted in long running queries which blocked subsequent reads. We dropped the index, as well as blocking database processes, and are now recovered.

Posted Mar 01, 2023 - 11:29 UTC

Resolved

This incident has been resolved.
Posted Mar 01, 2023 - 10:54 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Mar 01, 2023 - 10:53 UTC

Investigating

We're experiencing an elevated level of API errors and are currently looking into the issue.
Posted Mar 01, 2023 - 09:47 UTC
This incident affected: API (Root, Transactions) and Dashboard.