{"id":"IL-0141","org":"Google Cloud","date":"2023-11-02","docs":[],"title":{"en":"gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with \"Unable to sync sandbox\" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage","ko":"Workload Identity 를 쓰는 GKE 클러스터에서 gke-metadata-server 가 \"Unable to sync sandbox\" 디버그 로그를 대량으로 내 Cloud Logging 할당량을 소모하고 추가 요금이 나왔다"},"domain":["kubernetes","gcp","observability"],"impact":{"scope":"GKE clusters with Workload Identity enabled running gke-metadata-server 0.4.272 to 0.4.280: excess debug logs in Cloud Logging, consumed ingestion quota and excess billable usage; cluster functionality and performance not affected according to later updates","users":"unknown","region":"Taiwan (asia-east1), Hong Kong (asia-east2), Tokyo (asia-northeast1), Osaka (asia-northeast2), Seoul (asia-northeast3), Mumbai (asia-south1), Delhi (asia-south2), Singapore (asia-southeast1), Jakarta (asia-southeast2), Sydney (australia-southeast1), Melbourne (australia-southeast2), Warsaw (europe-central2), Finland (europe-north1), Madrid (europe-southwest1), Belgium (europe-west1), Berlin (europe-west10), Turin (europe-west12), London (europe-west2), Frankfurt (europe-west3), Netherlands (europe-west4), Zurich (europe-west6), Milan (europe-west8), Paris (europe-west9), Doha (me-central1), Dammam (me-central2), Tel Aviv (me-west1), Montréal (northamerica-northeast1), Toronto (northamerica-northeast2), São Paulo (southamerica-east1), Santiago (southamerica-west1), Iowa (us-central1), South Carolina (us-east1), Northern Virginia (us-east4), Columbus (us-east5), Dallas (us-south1), Oregon (us-west1), Los Angeles (us-west2), Salt Lake City (us-west3), Las Vegas (us-west4)"},"review":{"written_by":"session","reviewed_at":null,"reviewed_by":null},"lessons":null,"related":[],"sources":[{"url":"https://status.cloud.google.com/incidents/3UK9zjDHGXJ2oP2WboFR","kind":"official-status","license":"unknown","archived":null}],"summary":{"en":"gke-metadata-server is the GKE-managed component behind Workload Identity. Its versions 0.4.272 to 0.4.280 contained an incorrect configuration (the first detailed update called it a bug) that wrote debug logs containing \"Unable to sync sandbox\" at a high rate. Cloud Logging ingested those logs, so they consumed the customers' ingestion quota and created excess billable usage once the free monthly allotment was exceeded. The incident began at 10:45 US/Pacific on 2023-11-02, and the status page lists 39 Google Cloud locations as affected.\n\nGoogle had identified the root cause by the 14:09 update on 2023-11-02 and rolled out a fix across the GKE fleet that stopped the excess logs from being ingested into Cloud Logging; on 2023-11-08 the rollout was about 25% complete. Until the fix reached a cluster, customers on the GKE Rapid channel could upgrade the cluster control plane to 1.28.2-gke.1157000 or above, or 1.27.7-gke.1038000 or above, which carried a fixed gke-metadata-server. Customers on the Regular or Stable channels, or not using a release channel, had no workaround. The issue was resolved for all affected users as of 23:00 US/Pacific on 2023-11-09, and the incident was closed at 13:37 on 2023-11-10.","ko":"gke-metadata-server 는 GKE Workload Identity 기능을 이루는 GKE 관리 시스템 워크로드다. Workload Identity 는 GKE Standard 에서는 선택해서 켜고, GKE Autopilot 에서는 항상 켜져 있다. 이 서버의 0.4.272 부터 0.4.280 까지 버전에 잘못된 설정이 들어가 \"Unable to sync sandbox\" 디버그 로그를 높은 빈도로 냈다. 이 로그가 Cloud Logging 으로 수집되면서 수집 할당량을 소모했고, 월 무료 사용량을 넘긴 고객에게 추가 요금이 생겼다.\n\n장애는 2023-11-02 10:45 US/Pacific 에 시작됐다. Google 은 같은 날 원인을 찾았고, Rapid 채널 고객에게는 컨트롤 플레인을 1.28.2-gke.1157000 이상으로 올리면 고친 버전을 받을 수 있다고 안내했다. 11-06 에 전체 플릿에 수정을 배포하기 시작했다. 이 수정은 과도한 로그를 더 이상 Cloud Logging 으로 수집하지 않게 한다. 11-08 에 배포가 약 25% 진행됐다. Regular 채널, Stable 채널, 릴리스 채널을 쓰지 않는 고객에게는 그동안 우회 방법이 없었다.\n\n2023-11-09 23:00 US/Pacific 에 영향받은 모든 사용자의 문제가 해결됐고, 장애는 2023-11-10 13:37 에 종료로 기록됐다."},"trigger":{"en":"GKE clusters ran gke-metadata-server versions 0.4.272 to 0.4.280, which contained an incorrect configuration that wrote \"Unable to sync sandbox\" debug logs at a high rate.","ko":"gke-metadata-server 0.4.272 부터 0.4.280 까지의 버전에 잘못된 설정이 들어 있어 디버그 로그를 높은 빈도로 냈다. 11-02 공지는 이것을 버그라고 불렀다."},"severity":"unknown","symptoms":["log-flood","high-latency"],"timeline":{"detected_after":null,"resolved_after":"7d 12h 15m"},"date_kind":"incident","detection":{"en":"The first status update was posted at 13:32 US/Pacific on 2023-11-02, and the updates told customers how to check a cluster from the version annotation on the gke-metadata-server DaemonSet.","ko":"상태 페이지의 첫 공지는 2023-11-02 13:32 PDT 에 조사 중이라는 내용으로 올라왔다. 고객은 gke-metadata-server 데몬셋의 components.gke.io/component-version 어노테이션을 보고 자기 클러스터가 영향받는지 확인할 수 있었다."},"components":["credential-broker"],"root_cause":"software-bug","remediation":[{"text":{"en":"rolled out a fix across the GKE fleet so the excess logs were no longer ingested into Cloud Logging","ko":"과도한 로그를 더 이상 Cloud Logging 으로 수집하지 않게 하는 수정을 GKE 전체 플릿에 배포했다."}},{"text":{"en":"offered GKE Rapid channel customers a control plane upgrade to 1.28.2-gke.1157000 or above, or 1.27.7-gke.1038000 or above, as a workaround","ko":"Rapid 채널 고객에게 우회 방법으로 컨트롤 플레인을 1.28.2-gke.1157000 이상 또는 1.27.7-gke.1038000 이상으로 올리도록 안내했다."}}],"action_items":[],"contributing":[],"recurrence_of":null,"symptoms_text":{"en":"Google Kubernetes Engine (GKE) customers with Workload Identity enabled saw a high logging rate. Debug log lines containing the string \"Unable to sync sandbox\" were ingested into Cloud Logging, used up the Cloud Logging ingestion quota and, beyond the free monthly allotment, produced extra billable usage. Workload Identity is opt-in on GKE Standard and always enabled on GKE Autopilot.\n\nThe logs came from gke-metadata-server, a GKE-managed system workload that runs as a DaemonSet in kube-system and is part of Workload Identity. Clusters running gke-metadata-server 0.4.272 to 0.4.280 were affected. To check a cluster, Google gave the command kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml and the components.gke.io/component-version annotation in .spec.template.metadata.annotations: a version between 0.4.272 and 0.4.280 inclusive meant the cluster was affected.\n\nThe first detailed update, on 2023-11-02, said gke-metadata-server and the applications that depend on it kept working, but with some increased latency on metadata server calls and possibly reduced service, because gke-metadata-server was spending CPU on the excessive logging. From 2023-11-03 the updates said there was no impact on cluster functionality or performance and the only impact was extra logs in Cloud Logging.","ko":"Workload Identity 를 켠 GKE 클러스터에서 애플리케이션 로그 양이 크게 늘었다. 늘어난 로그는 \"Unable to sync sandbox\" 문자열이 든 디버그 로그였고, Cloud Logging 으로 수집돼 수집 할당량을 소모했다. 월 무료 사용량을 넘긴 고객에게는 추가 요금이 나왔다.\n\n로그를 낸 것은 GKE 가 관리하는 시스템 워크로드인 gke-metadata-server 였다. 영향받는 버전은 0.4.272 부터 0.4.280 까지였다. 클러스터가 영향받는지는 kubectl get daemonset -n kube-system -l k8s-app=gke-metadata-server -o yaml 로 데몬셋을 열어 .spec.template.metadata.annotations 의 components.gke.io/component-version 값을 보면 알 수 있다. 값이 0.4.272 와 0.4.280 사이면 영향받는 클러스터다.\n\n11-02 14:09 공지는 gke-metadata-server 와 이 서버에 의존하는 애플리케이션은 계속 동작하지만 메타데이터 서버 호출 지연이 조금 늘 수 있다고 했다. 공지는 과도한 로깅에 CPU 를 쓰느라 gke-metadata-server 의 성능이 떨어질 수 있다고도 했다. Cloud Logging 할당량을 다 쓸 수 있다는 경고도 있었다. 11-03 이후 공지는 클러스터 기능과 성능에는 영향이 없고, 영향은 Cloud Logging 에 쌓이는 추가 로그뿐이라고 적었다."},"updated_at":"2026-11-09T11:55:50.918137Z","url":"https://benchlet.com/incidentlib/app/cases/IL-0141","org_slug":"google-cloud"}