Cart

Cloud Status Dashboard

This page provides status information on the services that are part of Cloud Platform. Check back here to view the current status of the services listed below.

Google Compute Engine Incident #4933937362309678094

Global: pubsub.googleapis.com autoscaling not worked as expected

Incident began at 2021-12-07 01:56 and ended at 2021-12-13 16:59 (all times are US/Pacific).

Date Time Description
Dec 14, 2021 11:59

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) **Incident Start:** 07 Dec 2021 01:56 PST **Incident End:** 13 Dec 2021 16:59 PST **Duration:** 6 Days, 15 hours, 3 minutes **Affected Services and Features:** Google Compute Engine (GCE) Autoscaler - Autoscaling using the “pubsub.googleapis.com/subscription/num_undelivered_messages” metric. **Regions/Zones:** Global **Description:** GCE autoscaling service configured to use the “pubsub.googleapis.com/subscription/num_undelivered_messages” metric, experienced an issue while processing the autoscaling function. From preliminary analysis, the root cause was identified as code changes affecting the autoscaler policy validation that was misconfigured for the affected metric. **Customer Impact:** Customers using the metric pubsub.googleapis.com/subscription/num_undelivered_messages for autoscaling observed the autoscaling did not complete successfully. Autoscaling did not work for existing and new managed instance groups created by customers for the duration of the incident.

Dec 13, 2021 17:12

The issue with Google Compute Engine has been resolved for all affected users as of Monday, 2021-12-13 16:59 US/Pacific. We thank you for your patience while we worked on resolving the issue.

Dec 13, 2021 10:52

Summary: Global: pubsub.googleapis.com autoscaling not worked as expected Description: We believe the issue with Google Compute Engine is partially resolved for previously impacted projects. Newly created projects may still experience the issue until the rollout of the fix completes. Full resolution is expected to complete on Tuesday, 2021-12-14. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.

Dec 13, 2021 08:21

Summary: GCP engineer are investigating an issue with auto-scaling pubsub.googleapis.com may not be working as intended. Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Tuesday, 2021-12-14. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.

Dec 13, 2021 06:48

Summary: GCP engineer are investigating an issue with auto-scaling - pubsub.googleapis.com may not be working as intended. Description: Beginning at Monday, 2021-12-13 00:00 US/Pacific. We are experiencing an issue with Google Compute Engine beginning at Monday, 2021-12-13 05:21:05 PST. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-12-13 09:15 US/Pacific with current details. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.

Dec 13, 2021 06:26

Summary: GCP engineer are investigating an issue with auto-scaling - pubsub.googleapis.com may not be working as intended. Description: Beginning at Monday, 2021-12-13 00:00 US/Pacific. We are experiencing an issue with Google Compute Engine beginning at Monday, 2021-12-13 05:21:05 PST. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-12-13 07:15 US/Pacific with current details. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages may see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.

Dec 13, 2021 06:15

Summary: GCP engineer are investigating an issue with auto-scaling - pubsub.googleapis.com may not be working as intended. Description: Beginning at Monday, 2021-12-13 00:00 US/Pacific. We are experiencing an issue with Google Compute Engine beginning at Monday, 2021-12-13 05:21:05 PST. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2021-12-13 07:15 US/Pacific with current details. Diagnosis: Customers that are use scaling on the pubsub.googleapis.com/subscription/num_undelivered_messages masy see auto scaling not working. Workaround: As known workarounds, if customers have a filter with resource.label.subscription_id = ... they could change it to resource.labels.subscription_id = ... to make auto-scaling work.

  • All times are US/Pacific