This incident has been resolved. Our async processing uses GCP "spot" instances, and GCP unexpectedly started denying the creation of these spot instances in our kubernetes cluster. We have now created non-spot instances that will allow us to serve async tasks as normal, and have both spot and non-spot instances in place for future
Posted Jul 16, 2025 - 18:55 UTC
Monitoring
New node pools are up & running, tasks are running again. We are recovering.
Posted Jul 16, 2025 - 18:44 UTC
Identified
GCP is denying "spot instances" for our async processing kubernetes cluster. This means that no async tasks are being processed. We are creating a new node pool that does not rely on spot instances to get us back up & running
Posted Jul 16, 2025 - 18:41 UTC
Investigating
We are having issues with GCP provisioning compute instances for async processing. Async processing is impacted