{"id":"IL-0149","org":"Microsoft Azure","date":"2019-09-28","docs":[],"title":{"en":"Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy","ko":"2019-09-28 07:20 UTC 이전에 만든 VMSS 기반 AKS 클러스터의 업그레이드가 PlatformImageNotFound 오류로 실패했다."},"domain":["kubernetes","azure"],"impact":{"scope":"Upgrades of all AKS clusters on Virtual Machine Scale Sets created before 07:20 UTC on 28 September 2019","users":"unknown","region":"All regions"},"review":{"written_by":"session","reviewed_at":null,"reviewed_by":null},"lessons":null,"related":[],"sources":[{"url":"https://github.com/Azure/AKS/issues/1250","kind":"github","license":"unknown","archived":null}],"summary":{"en":"On 3 October 2019 AKS engineering opened a known-issue report: clusters using VMSS that were created before 07:20 UTC on 28 September 2019 failed to upgrade, because the node image the scale set referenced was reported as not available. The preliminary root cause was a recent service update that left instances of a backend service unhealthy, so requests could not complete. Customers were told not to upgrade these clusters until the issue was mitigated.\n\nCustomers could fix a cluster themselves with two az vmss update commands, setting VirtualMachineProfile.StorageProfile.ImageReference.version to \"2019.09.25\" and VirtualMachineProfile.StorageProfile.ImageReference.Sku to \"aks-ubuntu-1604-201909\" on the scale set in the MC resource group, and then running the upgrade again. Those who could not were asked to file a support request. The AKS and VMSS teams mitigated affected clusters by hand and rolled hot fixes out region by region; 14 regions were patched by 4 October, and the issue was closed on 15 October 2019 when all regions were patched.\n\nThe issue title says clusters created before 20 September, while its body gives 07:20 UTC on 28 September 2019.","ko":"2019-09-28 07:20 UTC 부터 Virtual Machine Scale Sets 를 쓰는 AKS 클러스터에서 업그레이드가 \"The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available.\" 오류로 실패했다. 이 시각 이전에 VMSS 로 만든 클러스터는 모든 리전에서 영향을 받았다. 예비 원인 분석에 따르면 최근 서비스 업데이트가 백엔드 서비스의 인스턴스를 비정상 상태로 만들었고, 그 때문에 요청이 완료되지 못했다.\n\nAKS 엔지니어링 팀은 2019-10-03 에 GitHub 이슈로 이를 알리며, 문제가 풀릴 때까지 해당 클러스터를 업그레이드하지 말라고 권했다. 직접 완화하려는 고객에게는 az vmss update 로 VMSS 이미지 참조의 version 을 \"2019.09.25\" 로, Sku 를 \"aks-ubuntu-1604-201909\" 로 바꾼 뒤 같은 업그레이드 명령을 다시 실행하라고 안내했다. 직접 해결하지 못한 고객은 Azure 지원 요청을 열면 지원팀과 당직 엔지니어가 클러스터를 고쳐 주기로 했다.\n\nAKS 팀과 VMSS 팀은 영향받은 클러스터를 손으로 완화하면서 장기 수정을 담은 릴리스를 리전마다 배포했다. 10월 4일 기준으로 14개 리전에 패치가 들어갔고, 10월 15일 모든 리전의 패치가 끝나 이슈가 닫혔다."},"trigger":{"en":"A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.","ko":"최근 서비스 업데이트가 백엔드 서비스의 인스턴스를 비정상 상태로 만들어 요청이 끝나지 못했다. 이 상태에서 고객이 클러스터 업그레이드를 시도했다."},"severity":"unknown","symptoms":["upgrade-stuck"],"timeline":{"detected_after":null,"resolved_after":null},"date_kind":"incident","detection":{"en":"Customers received failure messages when they tried to upgrade their clusters.","ko":"고객은 클러스터를 업그레이드하려 할 때 실패 알림을 받았다."},"components":["node-image","node-pool"],"root_cause":"deploy-rollout","remediation":[{"text":{"en":"published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again","ko":"az vmss update 명령으로 스케일 세트 이미지의 version 을 \"2019.09.25\" 로, Sku 를 \"aks-ubuntu-1604-201909\" 로 바꾼 뒤 업그레이드를 다시 실행하는 방법을 공개했다."}},{"text":{"en":"mitigated affected clusters by hand (AKS and VMSS teams)","ko":"AKS 팀과 VMSS 팀이 영향받은 클러스터를 손으로 완화했다."}},{"text":{"en":"rolled out hot fixes to all regions, completed by 15 October 2019","ko":"모든 리전에 핫픽스를 배포해 2019-10-15 에 마쳤다."}}],"action_items":[],"contributing":[],"recurrence_of":null,"symptoms_text":{"en":"From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: \"The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct.\" The issue title calls the error PlatformImageNotFound.\n\nAll VMSS-based clusters created before that time were affected, in all regions, and could not be upgraded without manual intervention. A customer asked in the issue how to tell whether a cluster was affected without trying an operation and having it fail; the issue gives no answer.","ko":"2019-09-28 07:20 UTC 부터 Virtual Machine Scale Sets 위의 AKS 클러스터에서 클러스터 업그레이드를 시도하면 실패 알림이 왔다.\n\n업그레이드 중 나온 오류 문구는 \"The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct.\" 였다. 이 시각 이전에 VMSS 로 만든 클러스터는 모든 리전에서 영향을 받았고, 손으로 개입하지 않으면 업그레이드할 수 없었다."},"updated_at":"2026-11-09T11:55:52.046960Z","url":"https://benchlet.com/incidentlib/app/cases/IL-0149","org_slug":"microsoft-azure"}