[{"id":"IL-0102","title":{"en":"CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch","ko":"배포와 워커 오토스케일링 폭주가 겹치자 크기를 고정해 둔 CoreDNS 가 메모리 한도를 넘겨 재시작된 뒤 회복하지 못했고, Buildkite 사이트 전체가 약 30분 멈췄다."},"org":"Buildkite","org_slug":"buildkite","date":"2026-08-25","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0102","shared_components":["node-pool","memory-limit","coredns"]},{"id":"IL-0001","title":{"en":"Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries","ko":"CoreDNS 파드가 메모리 한도 100Mi 에 걸려 OOM 으로 계속 죽으면서 쿠버네티스 클러스터 전체의 DNS 가 멈췄다"},"org":"Zalando","org_slug":"zalando","date":"2019-01-07","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0001","shared_components":["memory-limit","dns-resolver","coredns"]},{"id":"IL-0155","title":{"en":"Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes","ko":"NYC1 의 일부 DOKS 클러스터에서 공유 CPU 노드 위의 CoreDNS 때문에 DNS 가 간헐적으로 실패하고 NodeNotReady 가 났다"},"org":"DigitalOcean","org_slug":"digitalocean","date":"2026-07-09","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0155","shared_components":["node-pool","coredns"]},{"id":"IL-0048","title":{"en":"DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups","ko":"대형 EKS 클러스터에서 CoreDNS 가 VPC DNS 리졸버로 보내는 질의가 링크 로컬 트래픽의 초당 1024 패킷 한도에 걸려 DNS 조회가 간헐적으로 타임아웃됐다."},"org":"unknown","org_slug":"unknown","date":"2025-08-12","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0048","shared_components":["dns-resolver","coredns"]},{"id":"IL-0082","title":{"en":"Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse","ko":"EKS 노드 교체 중 CoreDNS 파드 하나가 종료되자 남은 CoreDNS 파드가 늘어난 부하를 버티지 못해 DNS 조회가 느려지고 타임아웃이 났다."},"org":"RebelMouse","org_slug":"rebelmouse","date":"2025-05-19","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0082","shared_components":["node-pool","coredns"]},{"id":"IL-0099","title":{"en":"User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes","ko":"메모리 상한 없이 8GB 만 보장한 프로필 때문에 워크숍 중 사용자 파드가 서로를 축출해 서버 시작이 15분을 넘고 커널이 죽었다."},"org":"2i2c","org_slug":"2i2c","date":"2025-05-12","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0099","shared_components":["node-pool","memory-limit"]},{"id":"IL-0169","title":{"en":"The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady","ko":"FR-PAR 의 Scaleway Kubernetes API 가 메모리 부족으로 쓰러져 노드 인증이 막히고 Kapsule 노드가 NotReady 가 됐다."},"org":"Scaleway","org_slug":"scaleway","date":"2024-01-24","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0169","shared_components":["memory-limit","kubelet"]},{"id":"IL-0016","title":{"en":"Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady","ko":"네임스페이스와 규칙이 늘어 live 클러스터의 Prometheus 가 메모리 부족으로 OOMKilled 되며 플랫폼 모니터링이 끊겼다"},"org":"UK Ministry of Justice","org_slug":"uk-ministry-of-justice","date":"2023-07-25","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0016","shared_components":["node-pool","memory-limit"]},{"id":"IL-0094","title":{"en":"Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub","ko":"JupyterHub 클러스터에서 학생 파드가 노드의 CPU 와 메모리를 다 써서 자원 보장이 부족했던 코어 파드가 자원을 얻지 못했고, 허브 전체가 멈췄다."},"org":"2i2c","org_slug":"2i2c","date":"2020-10-28","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0094","shared_components":["node-pool","memory-limit"]},{"id":"IL-0071","title":{"en":"Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots","ko":"작은 파일이 4만 개 넘게 든 단일 스테이지 이미지의 containerd 스냅샷이 쌓여 워커 노드의 inode 가 디스크보다 빠르게 소진되고 있었다"},"org":"Onkar Raghunath Shelke","org_slug":"onkar-raghunath-shelke","date":"2026-09-17","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0071","shared_components":["kubelet"]},{"id":"IL-0100","title":{"en":"PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries","ko":"노드가 파드의 CPU 요청에 비해 작아 CPU 가 포화되면서 PostHog 기능 플래그 서비스에 열흘 사이 장애가 세 번 났다."},"org":"PostHog","org_slug":"posthog","date":"2025-10-21","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0100","shared_components":["node-pool"]},{"id":"IL-0204","title":{"en":"An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server","ko":"Render 프랑크푸르트 클러스터에서 etcd 메모리 급증으로 컨트롤 플레인이 과부하에 걸렸고, 컨트롤 플레인 노드를 재부팅하자 그 위의 CoreDNS 가 다시 뜨지 못해 DNS 가 멈췄다."},"org":"Render","org_slug":"render","date":"2024-12-17","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0204","shared_components":["coredns"]},{"id":"IL-0044","title":{"en":"Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade","ko":"kube-prometheus-stack 업그레이드 뒤 Prometheus 가 WAL 재생 중 메모리를 50Gi 넘게 써서 시작할 때마다 OOMKilled 로 죽었다"},"org":"Saifeddine Rajhi","org_slug":"saifeddine-rajhi","date":"2024-06-14","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0044","shared_components":["memory-limit"]},{"id":"IL-0182","title":{"en":"Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation","ko":"트래픽이 급변할 때 수가 모자란 CoreDNS 가 과부하로 지연과 간헐적 네트워크 끊김을 일으켰고, 나중에는 과하게 늘린 CoreDNS 파드가 Karpenter 의 노드 통합을 막았다"},"org":"이언철","org_slug":"이언철","date":"2024-04-16","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0182","shared_components":["coredns"]},{"id":"IL-0110","title":{"en":"A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day","ko":"Grafana Agent 파드가 메모리를 많이 쓰던 노드에 Hangfire 파드가 함께 올라가 시스템 OOM 으로 CrashLoopBackOff 를 반복했고, 노드가 응답을 멈추고 NotReady 가 됐다."},"org":"DFDS","org_slug":"dfds","date":"2024-03-13","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0110","shared_components":["memory-limit"]},{"id":"IL-0165","title":{"en":"Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints","ko":"프랑크푸르트 LKE 의 백엔드 서비스가 메모리 부족으로 실패해 일부 사용자가 kubectl 로 kube-apiserver 에 접속하지 못했다"},"org":"Linode","org_slug":"linode","date":"2023-05-25","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0165","shared_components":["memory-limit"]},{"id":"IL-0070","title":{"en":"Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up","ko":"dask-gateway로 워커를 대량으로 띄우자 노드가 1분 만에 100대 늘었고, Prometheus가 메모리를 거의 다 쓰던 코어 노드에서 hub 파드를 비롯한 중요한 파드들이 축출되거나 재시작됐다"},"org":"2i2c","org_slug":"2i2c","date":"2023-02-01","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0070","shared_components":["node-pool"]},{"id":"IL-0206","title":{"en":"Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary","ko":"GKE 파드 안의 Consul 에이전트가 메모리 급증으로 거듭 OOM kill 되어 Rails 파드가 데이터베이스 복제본을 찾지 못했고, 모든 쿼리가 primary 로 몰려 GitLab.com 이 느려졌다."},"org":"GitLab","org_slug":"gitlab","date":"2022-04-25","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0206","shared_components":["memory-limit"]},{"id":"IL-0176","title":{"en":"Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999","ko":"시계열 급증으로 호스트 메모리가 바닥나자 oom_score_adj 999 를 받은 Prometheus 컨테이너가 커널 OOM Killer 에게 먼저 죽어 CrashLoopBackOff 에 빠졌다."},"org":"LINE","org_slug":"line","date":"2020-07-10","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0176","shared_components":["kubelet"]},{"id":"IL-0013","title":{"en":"The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets","ko":"노드 RAM 을 절반으로 줄이자 conntrack 테이블이 가득 차 집계 서비스에 연결 리셋과 오류가 늘었다"},"org":"loveholidays","org_slug":"loveholidays","date":"2020-02-15","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0013","shared_components":["node-pool"]},{"id":"IL-0015","title":{"en":"A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing","ko":"GKE 가 노드마다 IP 256개를 미리 잡아 /16 파드 주소 범위가 바닥나고 노드 증설과 배포가 멈췄다"},"org":"loveholidays","org_slug":"loveholidays","date":"2020-02-03","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0015","shared_components":["node-pool"]},{"id":"IL-0033","title":{"en":"Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds","ko":"kube-dns 파드 사이의 트래픽 불균형과 DNS 초당 패킷 한도 때문에 외부 DNS 조회가 많은 여러 서비스의 지연 시간이 20초까지 늘었다"},"org":"Airbnb","org_slug":"airbnb","date":"2019-11-21","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0033","shared_components":["dns-resolver"]},{"id":"IL-0026","title":{"en":"Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits","ko":"메모리 제한이 없는 fluentd-sumologic 파드가 메모리를 다 써서 운영 쿠버네티스 노드 두 대에 이틀 연속 SystemOOM 이 났다"},"org":"Blue Matador","org_slug":"blue-matador","date":"2019-06-15","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0026","shared_components":["memory-limit"]},{"id":"IL-0002","title":{"en":"CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit","ko":"빌더 파드가 노드 한 대에 몰리자 kubelet 이 기본 API QPS 한도에 막혀 파드 상태를 늦게 갱신했고, CI 빌드가 AWS IAM 자격 증명을 받지 못해 실패했다"},"org":"Zalando","org_slug":"zalando","date":"2019-06-11","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0002","shared_components":["kubelet"]},{"id":"IL-0040","title":{"en":"Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down","ko":"비운영 쿠버네티스 클러스터에 Helm 차트를 한꺼번에 설치하자 작은 gp2 루트 볼륨의 IOPS 가 바닥나, 워커 노드 세 대에서 kubelet 이나 Docker 가 내려가 세 대 모두 NotReady 가 됐다"},"org":"Saltside","org_slug":"saltside","date":"2017-04-26","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0040","shared_components":["kubelet"]}]