{"id":"IL-0030","org":"Zalando","date":"2019-02-11","docs":["https://github.com/kubernetes/kubernetes/issues/48638"],"title":{"en":"Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELB","ko":"쿠버네티스 v1.7.4 클러스터에서 노드들이 한꺼번에 NotReady 가 되어 쓸 수 있는 노드가 20% 미만으로 줄었다."},"domain":["kubernetes","aws"],"impact":{"scope":"one cluster; fewer than 20% of nodes available for about 15 minutes","users":"unknown","region":"unknown"},"review":{"written_by":"session","reviewed_at":null,"reviewed_by":null},"lessons":null,"related":[],"sources":[{"url":"https://www.slideshare.net/try_except_/lets-talk-about-failures-with-kubernetes-hamburg-meetup","kind":"talk","license":"unknown","archived":"https://web.archive.org/web/20241202203402/https://www.slideshare.net/try_except_/lets-talk-about-failures-with-kubernetes-hamburg-meetup"}],"summary":{"en":"Henning Jacobs of Zalando presented this as the first of five failure stories at a meetup in Hamburg on 2019-02-11. At the time Zalando counted 100 clusters and 373 accounts. This case covers only that first story; the CoreDNS outage from the same talk is a separate case.\n\nOn a cluster running Kubernetes v1.7.4 (CoreOS build), most nodes turned NotReady together, leaving fewer than 20% of nodes available. All of them recovered on their own after 15 minutes. The clues the team listed were that failure and recovery happened at the same time on every node, that the API server sat behind an AWS ELB, and that other users reported the same symptom, some of them about monthly.\n\nThe team matched it to the upstream issue kubernetes/kubernetes#48638, which was fixed in Kubernetes 1.8 and backported to 1.7.8.","ko":"이 사례는 Zalando 의 Henning Jacobs 가 2019-02-11 함부르크 밋업에서 발표한 쿠버네티스 장애 사례 가운데 첫 번째다. 쿠버네티스 v1.7.4+coreos.0 을 쓰는 클러스터에서 노드들이 동시에 NotReady 가 되어 쓸 수 있는 노드가 20% 미만으로 줄었다. 노드들은 15분 뒤 동시에 저절로 회복했다.\n\n팀은 API 서버가 AWS ELB 뒤에 있다는 점과 다른 사용자들도 같은 일을 겪고 있다는 점을 단서로 업스트림 이슈 kubernetes/kubernetes#48638 을 찾았다. 이 버그는 쿠버네티스 1.8 에서 고쳐졌고 1.7.8 로 백포트됐다."},"trigger":null,"severity":"unknown","symptoms":["node-not-ready"],"timeline":{"detected_after":null,"resolved_after":"15m"},"date_kind":"published","detection":null,"components":["kube-apiserver","cloud-load-balancer"],"root_cause":"software-bug","remediation":[{"text":{"en":"none; the nodes recovered on their own after about 15 minutes","ko":"노드들은 15분 뒤 저절로 복구됐다."}}],"action_items":[],"contributing":["known-bug-unpatched"],"recurrence_of":null,"symptoms_text":{"en":"Fewer than 20% of the nodes in one Zalando cluster were available. Most nodes reported NotReady at the same moment, and only a handful stayed Ready.\n\nThe kubectl get nodes output shown in the talk lists nodes such as ip-172-31-10-91...internal as NotReady, age 4d, version v1.7.4+coreos.0. In that excerpt five of eleven nodes are NotReady, four are Ready and two are Ready,SchedulingDisabled.\n\nThe nodes did not fail one by one. They all became unhealthy at the same time and all recovered at the same time, about 15 minutes later, without anyone intervening. Other Kubernetes users reported the same pattern, and some of them saw it roughly once a month.","ko":"클러스터에서 쓸 수 있는 노드가 전체의 20% 미만으로 줄었다. 노드들이 같은 시각에 한꺼번에 비정상이 됐다.\n\n발표 슬라이드에는 kubectl get nodes 출력 일부가 실려 있다. 이 출력에서 노드 11대 가운데 5대가 NotReady, 4대가 Ready, 2대가 Ready,SchedulingDisabled 였다. 노드 버전은 모두 v1.7.4+coreos.0 이었다.\n\n노드들은 같은 시각에 비정상이 되고 같은 시각에 회복했다. 클러스터는 15분 뒤 저절로 복구됐다.\n\n팀은 단서를 모으면서 API 서버가 AWS ELB 뒤에 있다는 점에 주목했다. 다른 사용자들도 같은 일을 겪고 있었고, 그중 일부는 거의 매달 이 일이 일어난다고 보고했다."},"updated_at":"2026-11-08T19:10:28.667255Z","url":"https://benchlet.com/incidentlib/app/cases/IL-0030","org_slug":"zalando"}