Database outage

Incident Report for Heron Data

Resolved

This incident has been resolved.
Posted Apr 07, 2023 - 11:33 UTC

Update

We are continuing to monitor for any further issues.
Posted Apr 07, 2023 - 10:59 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Apr 07, 2023 - 10:52 UTC

Identified

We have identified that a related database table is impacted, and are implementing a fix
Posted Apr 07, 2023 - 06:16 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Apr 07, 2023 - 06:04 UTC

Update

The database migration is complete and async processing is back online
Posted Apr 07, 2023 - 05:36 UTC

Update

We are running a backfill on the table in question which we believe will take ~6 hours, so we will provide another update around then
Posted Apr 06, 2023 - 22:55 UTC

Update

We have remediated part of the issue. Now the outage is limited to async processing and any route that involves fetching transaction categories (e.g., delete & get transactions, end user endpoints like /summary, /profit_and_loss, and various reports).

We are executing a fix which is underway for the remainder
Posted Apr 06, 2023 - 21:17 UTC

Update

We are continuing to work on a fix for this issue.
Posted Apr 06, 2023 - 19:42 UTC

Update

We are continuing to work on a fix for this issue.
Posted Apr 06, 2023 - 19:40 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Apr 06, 2023 - 19:17 UTC

Investigating

We are currently investigating an issue where we have reached the maximum number of IDs for a table in our database. We have to schedule some emergency downtime to resolve the issue
Posted Apr 06, 2023 - 19:16 UTC
This incident affected: API (Root, Transactions) and Dashboard, Website.