Cart

Cloud Status Dashboard

This page provides status information on the services that are part of Cloud Platform. Check back here to view the current status of the services listed below.

Google Compute Engine Incident #20009

We are experiencing an issue with Google Compute Engine instances running RHEL and CentOS 7 and 8. Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart.

Incident began at 2020-07-30 08:38 and ended at 2020-07-30 13:17 (all times are US/Pacific).

Date Time Description
Jul 30, 2020 13:17

The issue with Google Compute Engine instances running RHEL and CentOS 7 and 8 is being actively investigated by Redhat. More details on this issue are available in the following article and bugs: - https://access.redhat.com/solutions/5272311 - https://bugzilla.redhat.com/show_bug.cgi?id=1861977 (RHEL 8) - https://bugzilla.redhat.com/show_bug.cgi?id=1862045 (RHEL 7) - https://issuetracker.google.com/162523000 Please follow our public issue tracker posting (https://issuetracker.google.com/162523000) for updates on this issue going forward. No further updates will be provided here.

Jul 30, 2020 10:52

Description: We are experiencing an issue with Google Compute Engine instances running RHEL and CentOS 7 and 8. More details on this issue are available in the following article and bugs: https://access.redhat.com/solutions/5272311 https://bugzilla.redhat.com/show_bug.cgi?id=1861977 (RHEL 8) https://bugzilla.redhat.com/show_bug.cgi?id=1862045 (RHEL 7) Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP". This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: `rpm -q shim-x64` Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64 Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run `yum downgrade shim\* grub2\* mokutil` to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command. We will provide an update by Thursday, 2020-07-30 14:00 US/Pacific with current details. Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP". This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: `rpm -q shim-x64` Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64 Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run `yum downgrade shim\* grub2\* mokutil` to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.

Jul 30, 2020 09:52

Description: We are experiencing an issue with Google Compute Engine RHEL and CentOS 7 and 8 instances. Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP". This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: `rpm -q shim-x64` Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64 Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run `yum downgrade shim\* grub2\* mokutil` to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2020-07-30 11:00 US/Pacific with current details. Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP". This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: `rpm -q shim-x64` Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64 Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run `yum downgrade shim\* grub2\* mokutil` to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.

Jul 30, 2020 09:41

Description: We are experiencing an issue with Google Compute Engine RHEL and CentOS 7 and 8 instances. Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart. Customers should not update or reboot instances running RHEL or CentOS 7 and 8. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2020-07-30 10:30 US/Pacific with current details. Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP". This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64 Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64 Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run "yum downgrade shim\* grub2\* mokutil" to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.

Jul 30, 2020 09:27

Description: We are experiencing an issue with Google Compute Engine RHEL and CentOS 7 and 8 instances. Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart. Customers should not update or reboot instances running RHEL or CentOS 7 and 8. Our engineering team continues to investigate the issue. We are also preparing instructions for recovery from this issue for affected instances. We will provide an update by Thursday, 2020-07-30 10:15 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart. Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8.

Jul 30, 2020 08:49

Description: We are experiencing an issue with Google Compute Engine RHEL and CentOS 7 and 8 instances. Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart. Customers should not update or reboot instances running RHEL or CentOS 7 and 8. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2020-07-30 09:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart. Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8.

  • All times are US/Pacific