Cart

Cloud Status Dashboard

This page provides status information on the services that are part of Cloud Platform. Check back here to view the current status of the services listed below.

Google Cloud Storage Incident #13628602836948027359

us multiregion: Elevated tail latency on read operations for GCS buckets

Incident began at 2021-12-17 09:10 and ended at 2021-12-17 22:11 (all times are US/Pacific).

Date Time Description
Dec 20, 2021 10:04

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) **Incident Start:** 17 December 2021 09:10 **Incident End:** 17 December 2021 22:11 **Incident Duration:** 13 hours, 1 minute **Affected Services and Features:** Google Cloud Storage (GCS) - Read and Write object operations **Regions/Zones:** US multi-region **Description:** Google Cloud Storage buckets experienced elevated tail latency on read and write object operations for a total duration of 13 hours, 1 minute. From preliminary analysis, an internal job incorrectly placed a significant load on the backend database. This originally presented as increased tailed latency. The database was significantly overloaded leading to elevated 5xx errors for a 47 minute period between 10:50 and 11:37. The 5XX errors were resolved at 11:37 by adding additional resources and stopping the offending job. However, the job caused an extended backlog of work, which took until 22:11 to clear. **Customer Impact:** * Elevated 5xx errors related to read timeouts between 10:50 and 11:37 * Elevated tail latency on read operations from GCS buckets for the duration of the incident.

Dec 17, 2021 12:49

The issue with Google Cloud Storage has been resolved for all affected users as of Friday, 2021-12-17 11:36 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Dec 17, 2021 11:47

Summary: us multiregion: Elevated tail latency on read operations for GCS buckets Description: We believe the issue with Google Cloud Storage is partially resolved and the error rates decreased significantly. We do not have an ETA for full resolution at this point. We will provide an update by Friday, 2021-12-17 13:30 US/Pacific with current details. Diagnosis: Affected customers may experience elevated tail latency on read operations for buckets in the us multiregion. Workaround: Retrying failed or slow requests may succeed due to the relatively low error rate.

Dec 17, 2021 11:22

Summary: us multiregion: Elevated tail latency on read operations for GCS buckets Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Friday, 2021-12-17 13:00 US/Pacific. Diagnosis: Affected customers may experience elevated tail latency on read operations for buckets in the us multiregion. Workaround: Retrying failed or slow requests may succeed due to the relatively low error rate.

Dec 17, 2021 11:00

Summary: us: Elevated latency in us GCS buckets Description: We are experiencing an issue with Google Cloud Storage beginning at Friday, 2021-12-17 09:00 US/Pacific. Our engineering team continues to investigate the issue. Current estimates indicates that 1% of GCS requests are impacted. We will provide an update by Friday, 2021-12-17 13:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers may experience elevated tail latency on their buckets in the us multiregion. Workaround: None at this time.

Dec 17, 2021 10:43

Summary: us: Elevated latency in us GCS buckets Description: We are investigating a potential issue with Google Cloud Storage. We will provide more information by Friday, 2021-12-17 11:56 US/Pacific. Diagnosis: Affected customers may experience elevated latency on their buckets in the us multiregion. Workaround: None at this time.

  • All times are US/Pacific