{"id":"IL-0130","org":"Google Cloud","date":"2024-10-22","docs":[],"title":{"en":"Deployments of large models from Vertex Model Garden failed with \"Model server never became ready\" in most regions for 13 days after a storage provisioning configuration change","ko":"내부 스토리지 프로비저닝 설정 오류로 Vertex Model Garden 의 100GB 넘는 대형 모델 배포가 13일 넘게 \"Model server never became ready\" 오류를 내며 실패했다."},"domain":["gcp","ai-infra","kubernetes"],"impact":{"scope":"Deployments of large models (more than 100 GB of disk) in Vertex AI Online Prediction (Vertex Model Garden deployments); status updates name L4 GPUs and GKE Autopilot clusters","users":"Customers deploying large Vertex Model Garden models","region":"All regions except asia-southeast1, europe-west4, us-central1, us-east1, us-east4"},"review":{"written_by":"session","reviewed_at":null,"reviewed_by":null},"lessons":{"en":"A configuration error that affects only one class of deployments, here models needing more than 100 GB of disk, can stay in place across most regions for almost two weeks. The visible symptom, a model server that never becomes ready, does not point at storage provisioning, and the fix was to roll the configuration back. The report lists no follow-up actions.","ko":"대형 모델 배포처럼 일부 조건에서만 실패하는 설정 오류가 대부분 리전에서 13일 넘게 이어졌다. 장애는 13일 가까이 지난 뒤에야 상태 페이지에 올라왔고, 원인이 된 설정 변경을 롤백하자 해결됐다."},"related":[],"sources":[{"url":"https://status.cloud.google.com/incidents/2GcJSJ2mZp3xw91PbJ87","kind":"official-status","license":"unknown","archived":null}],"summary":{"en":"Vertex AI Online Prediction serves models that customers deploy from Vertex Model Garden. For 13 days, 6 hours and 22 minutes, from 22 October 2024 at 14:08 to 4 November 2024 at 20:30 US/Pacific, deployments of large models, meaning models that need more than 100 GB of disk, failed in most regions with \"Model server never became ready\". The status updates tied the failures to deployments on L4 GPUs and on GKE Autopilot clusters.\n\nGoogle's preliminary analysis placed the root cause in an internal storage provisioning configuration error that was implemented as part of a recent change. Engineers mitigated the problem by rolling that configuration change back, and the issue was resolved for all affected users as of 20:28 US/Pacific on 4 November 2024. Until then customers could work around it by deploying in one of the five unaffected regions.","ko":"Google Cloud 의 Vertex AI Online Prediction 에서 디스크가 100GB 넘게 필요한 대형 모델의 Vertex Model Garden 배포가 대부분 리전에서 실패했다. 기간은 2024년 10월 22일 14:08 부터 11월 4일 20:30 US/Pacific 까지로, 최대 13일 6시간 22분이었다. 고객은 배포할 때 \"Model server never became ready\" 오류를 받았다.\n\n예비 분석 결과 원인은 최근 변경과 함께 들어간 내부 스토리지 프로비저닝 설정 오류였다. Google 엔지니어들은 문제를 일으킨 설정 변경을 롤백해 영향을 없앴고, 11월 4일 20:28 US/Pacific 에 모든 고객의 문제가 해결됐다. 장애 기간 동안 고객은 영향을 받지 않은 리전에 배포하는 방법으로 피해 갈 수 있었다."},"trigger":{"en":"A recent change introduced an internal storage provisioning configuration error.","ko":"최근 변경과 함께 들어간 내부 스토리지 프로비저닝 설정에 오류가 있었다."},"severity":"unknown","symptoms":["deploys-failing"],"timeline":{"detected_after":"12d 23h 9m","resolved_after":"13d 6h 22m"},"date_kind":"incident","detection":{"en":"The incident began on 22 October 2024 at 14:08 but was first posted on the status page on 4 November 2024 at 13:17 US/Pacific, about 13 days later. The first post said large Vertex Model Garden deployments (greater than 100GB) on a GKE Autopilot cluster might fail.","ko":"장애는 2024년 10월 22일 14:08 에 시작됐지만 상태 페이지에 처음 올라온 것은 약 13일 뒤인 11월 4일 13:17 US/Pacific 이었다. 첫 게시는 GKE Autopilot 클러스터로 가는 100GB 넘는 Vertex Model Garden 배포가 실패할 수 있다고 적었다."},"components":[],"root_cause":"config-change","remediation":[{"text":{"en":"rolled back the configuration change that introduced the storage provisioning error","ko":"Google 엔지니어들이 문제를 일으킨 설정 변경을 롤백했다."}}],"action_items":[],"contributing":[],"recurrence_of":null,"symptoms_text":{"en":"From 22 October 2024 at 14:08 US/Pacific, customers deploying large models, those that need more than 100 GB of disk, through Vertex AI Online Prediction (Vertex Model Garden deployments) had their deployments fail with the error \"Model server never became ready\".\n\nThe failures happened in most regions. Deployments in asia-southeast1, europe-west4, us-central1, us-east1 and us-east4 were not affected. Status updates on 4 November described the failing deployments as large Model Garden deployments on L4 GPUs, and one update as large deployments (greater than 100GB) on a GKE Autopilot cluster.\n\nThe incident first appeared on the status page on 4 November 2024 at 13:17 US/Pacific, almost 13 days after it began, and the updates that day listed no workaround. The incident report published afterwards said customers had been able to deploy in one of the unaffected regions instead.","ko":"2024년 10월 22일 14:08 US/Pacific 부터 Vertex AI Online Prediction 에서 대형 모델을 배포하던 고객은 \"Model server never became ready\" 오류를 받았다. 디스크가 100GB 넘게 필요한 모델이 대상이었다.\n\n장애는 asia-southeast1, europe-west4, us-central1, us-east1, us-east4 를 뺀 대부분 리전에서 났다. 상태 페이지는 11월 4일에야 문제를 알렸다. 첫 게시는 GKE Autopilot 클러스터로 가는 100GB 넘는 Vertex Model Garden 배포가 실패할 수 있다고 알렸고, 뒤이은 게시는 L4 GPU 를 쓰는 대형 배포가 실패할 수 있다고 적었다. 그때까지는 우회 방법이 없다고 안내했다."},"updated_at":"2026-11-09T11:55:49.391219Z","url":"https://benchlet.com/incidentlib/app/cases/IL-0130","org_slug":"google-cloud"}