Cart

Cloud Status Dashboard

This page provides status information on the services that are part of Cloud Platform. Check back here to view the current status of the services listed below.

Google Compute Engine Incident #20001

The issue with instance DELETE and STOP operations in Google Compute Engine has been mitigated.

Incident began at 2020-01-28 12:18 and ended at 2020-01-29 16:26 (all times are US/Pacific).

Date Time Description
Jan 29, 2020 16:26

The issue with Google Compute Engine operations in us-central1-a has been resolved for all affected projects as of Wednesday, 2020-01-29 16:16 US/Pacific. We thank you for your patience while we’ve worked on resolving the issue.

Jan 29, 2020 14:09

Description: Between Wednesday, 2020-01-29 04:00 US/Pacific and 07:30 US/Pacific instance stop and delete operations may have failed or appeared stuck in us-central1-a. As of 07:30 US/Pacific stop and delete operations are succeeding and we are now focusing on ensuring that the issue does not recur and working through the backlog of operations. Running instances were not impacted. Other zones in us-central1 were not impacted. New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again. Mitigation efforts continue to accelerate the backlog processing. Products which depend on GCE may also have been unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore. We will provide an update by Wednesday, 2020-01-29 16:30 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again.

Jan 29, 2020 10:52

Description: Between Wednesday, 2020-01-29 04:00 US/Pacific and 07:30 US/Pacific instance stop and delete operations may have failed or appeared stuck in us-central1-a. As of 07:30 US/Pacific stop and delete operations are succeeding and we are now focusing on ensuring that the issue does not recur and working through the backlog of operations. Running instances were not impacted. Other zones in us-central1 were not impacted. Products which depend on GCE may also have been unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore. We will provide an update by Wednesday, 2020-01-29 14:30 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again.

Jan 29, 2020 09:56

Description: We are experiencing an issue with stopping and deleting Google Compute Engine instances affecting mainly us-central1-a, with a subset of global operations impacted as well, the majority of which are projects.setCommonInstanceMetadataInstance operations. Stop and delete operations between Wednesday, 2020-01-29 04:00 US/Pacific and 09:00 US/Pacific may have failed or appear stuck. Stop and delete operations created after 09:00 US/Pacific should complete as expected, even if they had previously failed or are stuck. Mitigation is underway and we are currently working through the backlog of operations. Currently running instances are not impacted. Other zones in us-central1 are not impacted. Products which depend on GCE may also be unable to stop or delete jobs. Impacted products include GKE, Dataflow, Dataproc, CloudSQL, Memorystore We will provide an update by Wednesday, 2020-01-29 11:00 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Setting projects.setCommonInstanceMetadata may also fail or complete with an error, this is a global operation. Workaround: New requests should be successful. If there are stuck operations that need to be completed urgently, please try your request again.

Jan 29, 2020 08:44

Description: We are experiencing an issue with stopping and deleting Google Compute Engine instances affecting mainly us-central1-a, with a subset of global operations impacted as well, the majority of which are projects.setCommonInstanceMetadataInstance operations. Impact began Wednesday, 2020-01-29 06:02 US/Pacific. Mitigation is underway and we are currently working through the backlog operations. Currently running instances are not impacted. We will provide an update by Wednesday, 2020-01-29 10:00 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error in us-central1-a. Errors may be in the form of “rate limit exceeded”. Setting projects.setCommonInstanceMetadata may also fail or complete with an error, this is a global operation. Workaround: None at this time.

Jan 29, 2020 07:35

Description: We are experiencing an issue with Google Compute Engine affecting mainly us-central1-a, although other zones in the region may see the issue as well, though at a much lower rate. Instance STOP and DELETE requests may fail or complete with an error, beginning Wednesday, 2020-01-29 06:02 US/Pacific. Symptoms: Instance STOP and DELETE requests may fail or complete with an error. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2020-01-29 08:45 US/Pacific with current details. Diagnosis: Instance STOP and DELETE requests may fail or complete with an error. Workaround: None at this time.

  • All times are US/Pacific