Cart

Cloud Status Dashboard

This page provides status information on the services that are part of Cloud Platform. Check back here to view the current status of the services listed below.

Cloud Spanner Incident #8484543035619786647

us-central1-a,c: Cloud spanner jobs reporting high read latencies. Incident mitigated.

Incident began at 2021-10-06 13:50 and ended at 2021-10-06 17:42 (all times are US/Pacific).

Date Time Description
Oct 07, 2021 10:33

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support (All Times US/Pacific) **Incident Start:** 06 October 2021 13:50 **Incident End:** 06 October 2021 17:42 **Duration:** 3 hours, 52 minutes **Affected Services and Features:** - Cloud Spanner: Latency **Regions/Zones:** us-central1 **Description:** Cloud Spanner experienced elevated p99.9 read latencies in us-central1 in some limited cases for a duration of 3 hours, 52 minutes. From preliminary analysis, the root cause of the issue was a resource contention issue on a small portion of the Spanner frontend servers in us-central1 that affected p99.9 latency for some customers. **Customer Impact:** A small portion of the regional Cloud Spanner instances in us-central1 experienced increased p99.9 latency for read requests. Multi-regional Cloud Spanner instances (nam6, nam7, nam9, nam10, nam11, nam12, and nam-eur-asia1) where the requests were routed to the impacted frontend servers in us-central1 were affected similarly. **Additional details:** The issue has been fully mitigated by isolating the impacted frontend servers, and we do not believe there is a risk of recurrence at this time.

Oct 06, 2021 18:28

The issue with Cloud Spanner has been resolved for all affected projects as of Wednesday, 2021-10-06 17:42 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Oct 06, 2021 17:51

Summary: us-central1-a,c: Cloud spanner jobs reporting high read latencies. Investigations ongoing. Description: We are experiencing an issue with high read latencies on Cloud Spanner jobs at us-central1-a,c beginning at Wednesday, 2021-10-06 14:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2021-10-06 18:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: None at this time. Workaround: None at this time.

  • All times are US/Pacific