GCP outage

Incident Report for Heron Data

Postmortem

Google Cloud Platform had a system outage that impacted our systems: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW . This mainly impacted our ability to send data to GCP such as storing files or other information

All data sent via email-based integrations have been recovered (emails / files / end users have been reprocessed) and no data was lost during the outage. Any API-based requests that Heron indicated were successful during this time were indeed successful, but customers may have experienced a higher-than-normal error rate during this time.

We are pleased to report that no data was lost during this outage, and have made some changes in the Heron system to make it easier to recover from similar outages in the future.

Posted Jun 13, 2025 - 19:54 UTC

Resolved

This incident has been resolved.
Posted Jun 13, 2025 - 07:53 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jun 12, 2025 - 22:50 UTC

Identified

GCP has advised that they are experiencing varying degrees of impact on different components. We will continue to monitor and recover as possible https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
Posted Jun 12, 2025 - 20:03 UTC

Monitoring

GCP services have recovered. Our systems are recovering
Posted Jun 12, 2025 - 19:54 UTC

Update

Read requests are succeeding, but post requests are failing
Posted Jun 12, 2025 - 18:09 UTC

Update

We are continuing to investigate this issue.
Posted Jun 12, 2025 - 18:08 UTC

Investigating

We are experiencing issues with one of our cloud providers, GCP. Namely, we are seeing degradation in the cloud storage, cloud run, and AI services
Posted Jun 12, 2025 - 18:08 UTC
This incident affected: API (Root, Transactions).