Cart

Cloud Status Dashboard

This page provides status information on the services that are part of Cloud Platform. Check back here to view the current status of the services listed below.

Cloud Firestore Incident #7467022363546712602

High latency and increased error rates in northamerica multi-region.

Incident began at 2021-06-22 19:38 and ended at 2021-06-22 20:51 (all times are US/Pacific).

Date Time Description
Jun 23, 2021 14:40

**Incident Start:** 22 June 2021 19:38 **Incident End:** 22 June 2021 20:51 **Duration:** 1 hours, 13 minutes **Affected Services and Features:** Google Cloud Firestore **Regions/Zones:** nam5 multi-region **Description:** Google Cloud Firestore experienced elevated latency and error rates on up top 50% of projects in the nam5 multi-region for a duration of 1 hour and 13 minutes. The root cause of the issue was a traffic spike from an internal daily batch job which performs quota adjustments. This caused lock contention on the backend databases which, in conjunction with poor traffic distribution and misdirected traffic, ultimately overloaded the service. **Customer Impact:** - Affected customers may see “Deadline Exceeded” errors and degraded performance.

Jun 22, 2021 21:53

The issue with Cloud Firestore has been resolved for all affected users as of Tuesday, 2021-06-22 21:53 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Jun 22, 2021 21:26

Summary: High latency and increased error rates in northamerica multi-region. Description: We are experiencing an intermittent issue with Cloud Firestore beginning at Tuesday, 2021-06-22 19:38 US/Pacific. At the moment we see performance improvements and error rate is back to its original state. Our engineering team continues to monitor the situation. We will provide an update by Tuesday, 2021-06-22 22:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers might see 'Deadline Exceeded' errors and degraded performance. Workaround: None at this time.

Jun 22, 2021 20:45

Summary: High latency and increased error rates in northamerica multi-region. Description: We are experiencing an intermittent issue with Cloud Firestore beginning at Tuesday, 2021-06-22 19:38 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2021-06-22 21:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers might see 'Deadline Exceeded' errors and degraded performance. Workaround: None at this time.

  • All times are US/Pacific