{"id":"IL-0097","org":"2i2c","date":"2026-03-31","docs":["https://github.com/2i2c-org/infrastructure/pull/8008","https://github.com/2i2c-org/infrastructure/blob/main/.github/workflows/recurrent-k8s-aws-upgrades.yaml"],"title":{"en":"A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created","ko":"cluster autoscaler 버전을 올린 뒤 Kubernetes 1.33 에 남아 있던 클러스터에서 스케일업이 일어나지 않아 새 노드가 만들어지지 않았다"},"domain":["kubernetes","aws"],"impact":{"scope":"berkeley-geojupyter cluster, prod hub: new nodes were not created and servers failed to start","users":"unknown; the alert was triggered by an automatic health check run, not by users who could not start servers","region":"unknown"},"review":{"written_by":"session","reviewed_at":null,"reviewed_by":null},"lessons":{"en":"Because the context of the autoscaler bump was fresh, the team went down the right path from the start. The upgrade batches had been designed with only 1.32 clusters in mind, so the one 1.33 cluster was missed and got an incompatible autoscaler.\n\n2i2c notes it was lucky that an automatic health check, not real users, hit the failure first.","ko":"대응자들이 autoscaler 버전을 올린 맥락을 생생히 기억하고 있어서 처음부터 올바른 방향으로 조사했다. 반면 팀이 업그레이드 묶음을 설계할 때 1.32 클러스터만 염두에 둬서 1.33 클러스터가 빠졌다. 이번에 맞춘 autoscaler 9.55.0 은 Kubernetes 1.34 용 버전이었다."},"related":[],"sources":[{"url":"https://github.com/2i2c-org/incident-reports/blob/main/reports/2026-03-31-k8s-autoscaler-version-incompatibity-berkeley-geojupyter.pdf","kind":"github","license":"unknown","archived":null}],"summary":{"en":"The berkeley-geojupyter cluster ran an older Kubernetes version, 1.33, than most of 2i2c's clusters. When the upgrade batches were designed, only clusters running 1.32 were taken into account, so this cluster was missed. A day before the incident an upgrade bumped the cluster autoscaler version, and the new autoscaler did not work with Kubernetes 1.33: it triggered no scale-up events, so no new nodes were spawned and servers on the prod hub failed to start.\n\nBecause the context of the autoscaler bump was fresh, the team went down the right path from the start. At 15:41 the team decided to upgrade the cluster to Kubernetes 1.34, and then downgraded the cluster autoscaler by one patch version to 9.55.0, the version meant for Kubernetes 1.34. At 16:48 the team confirmed this fixed it. The report gives the impact time as 15:10 to 17:00 and the duration as 1h 49m 10s.","ko":"2i2c 의 berkeley-geojupyter 클러스터는 AWS 에서 돌며 prod 허브를 운영한다. 이 클러스터는 대다수 클러스터보다 오래된 Kubernetes 1.33 을 쓰고 있었다. 하루 전 cluster autoscaler 버전을 올리는 업그레이드가 머지되면서 클러스터 버전과 autoscaler 버전이 맞지 않게 됐다. autoscaler 가 스케일업을 하지 않아 새 노드가 만들어지지 않았고, 그래서 서버가 시작되지 못했다.\n\n팀은 업그레이드 묶음을 설계할 때 1.32 를 쓰는 클러스터만 고려했고, 1.33 을 쓰던 이 클러스터는 빠졌다. 팀은 클러스터를 1.33 에서 1.34 로 올리고, autoscaler 를 한 패치 버전 내려 k8s 1.34 용인 9.55.0 으로 맞췄다. 16:48 에 문제가 풀렸다. 영향 시간은 15:10 부터 17:00 까지였고, 지속 시간은 1시간 49분 10초로 기록됐다."},"trigger":{"en":"An upgrade merged on 2026-03-30 bumped the cluster autoscaler version, and the new version was incompatible with Kubernetes 1.33, which the berkeley-geojupyter cluster still ran.","ko":"하루 전에 머지한 업그레이드에서 cluster autoscaler 버전이 올라갔다. berkeley-geojupyter 클러스터는 대다수 클러스터보다 오래된 Kubernetes 1.33 을 쓰고 있었고, 새 autoscaler 버전과 맞지 않았다."},"severity":"unknown","symptoms":["autoscaler-scale-up-failed"],"timeline":{"detected_after":null,"resolved_after":"1h 49m"},"date_kind":"incident","detection":{"en":"An alert that at least two servers failed to start in the last 30 minutes fired at 15:10 and opened PagerDuty incident #2057. The report notes it was triggered by an automatic health check run, not by users being unable to start servers.","ko":"15:10 에 자동 상태 점검이 서버 시작 실패를 잡아 PagerDuty 알림 \"[FIRING:1] At least two servers failed to start in the last 30m\" 을 API 로 띄웠고, 장애 #2057 이 열렸다. 실제 사용자가 서버를 못 띄워 알게 된 것이 아니었다."},"components":["cluster-autoscaler"],"root_cause":"upgrade-breaking-change","remediation":[{"text":{"en":"upgraded the cluster's Kubernetes version from 1.33 to 1.34","ko":"클러스터의 Kubernetes 버전을 1.33 에서 1.34 로 올렸다."}},{"text":{"en":"downgraded the cluster autoscaler by one patch version to 9.55.0, the version meant for Kubernetes 1.34","ko":"cluster autoscaler 를 한 패치 버전 내려 k8s 1.34 용인 9.55.0 으로 맞췄다."}}],"action_items":[{"text":{"en":"The change in https://github.com/2i2c-org/infrastructure/pull/8008 (the report gives only the link)","ko":"2i2c-org/infrastructure 저장소의 PR 8008 로 후속 수정을 한다."}},{"text":{"en":"Use recurrent issues, such as the recurrent-k8s-aws-upgrades workflow, to standardize Kubernetes versions across clusters","ko":"recurrent-k8s-aws-upgrades.yaml 워크플로가 만드는 정기 업그레이드 이슈를 통해 클러스터마다 다른 Kubernetes 버전을 하나로 맞춘다."}}],"contributing":["skipped-upgrades","assumption-unverified"],"recurrence_of":null,"symptoms_text":{"en":"At 15:10 (Bucharest time) on 2026-03-31 an alert fired for the prod hub on the berkeley-geojupyter cluster on AWS: \"[FIRING:1] At least two servers failed to start in the last 30m cluster=berkeley-geojupyter hub=prod (provider=aws immediate action needed)\". It opened PagerDuty incident #2057.\n\nSix minutes later an engineer saw no scale-up event for the cluster and started checking the node group labels. New nodes were not being created because the cluster autoscaler was not triggering scale-up events. Another engineer noticed that the cluster was still running Kubernetes 1.33, and the autoscaler deployment logs were checked next because the autoscaler did not seem to behave properly.\n\nAt 15:37 the engineer who had merged an upgrade the day before connected the failure to it: that upgrade had also bumped the cluster autoscaler version.","ko":"2026-03-31 15:10 부쿠레슈티 시각에 PagerDuty 알림 \"[FIRING:1] At least two servers failed to start in the last 30m cluster=berkeley-geojupyter hub=prod (provider=aws immediate action needed)\" 이 울렸다. 이 알림은 서버를 못 띄운 사용자의 신고가 아니라 자동 상태 점검에서 나왔다.\n\n15:16 에 대응자는 스케일업 이벤트가 보이지 않는다고 적고 노드 그룹 라벨부터 확인했다. 15:23 에는 이 클러스터가 왜 아직 1.33 을 쓰고 있느냐는 의문이 나왔다.\n\n15:35 에 대응자는 autoscaler 가 제대로 동작하지 않는다고 보고 autoscaler Deployment 로그를 보기 시작했다. 15:37 에 대응자는 전날 머지한 업그레이드에서 autoscaler 버전도 함께 올라갔다는 것을 떠올렸다."},"updated_at":"2026-11-10T00:00:32.827732Z","url":"https://benchlet.com/incidentlib/app/cases/IL-0097","org_slug":"2i2c"}