[{"id":"IL-0006","title":{"en":"VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion","ko":"AWS VPC CNI 가 자원이 찬 노드에 IP 를 붙잡아 두어 us-east-1 서브넷 IP 가 바닥났고, WARM_IP_TARGET 을 바꾼 조치가 파드 시작 실패를 더 오래 끌었다"},"org":"Neon","org_slug":"neon","date":"2025-05-16","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0006","shared_components":["cluster-autoscaler","subnet"]},{"id":"IL-0102","title":{"en":"CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch","ko":"배포와 워커 오토스케일링 폭주가 겹치자 크기를 고정해 둔 CoreDNS 가 메모리 한도를 넘겨 재시작된 뒤 회복하지 못했고, Buildkite 사이트 전체가 약 30분 멈췄다."},"org":"Buildkite","org_slug":"buildkite","date":"2026-08-25","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0102","shared_components":["node-pool"]},{"id":"IL-0155","title":{"en":"Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes","ko":"NYC1 의 일부 DOKS 클러스터에서 공유 CPU 노드 위의 CoreDNS 때문에 DNS 가 간헐적으로 실패하고 NodeNotReady 가 났다"},"org":"DigitalOcean","org_slug":"digitalocean","date":"2026-07-09","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0155","shared_components":["node-pool"]},{"id":"IL-0100","title":{"en":"PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries","ko":"노드가 파드의 CPU 요청에 비해 작아 CPU 가 포화되면서 PostHog 기능 플래그 서비스에 열흘 사이 장애가 세 번 났다."},"org":"PostHog","org_slug":"posthog","date":"2025-10-21","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0100","shared_components":["node-pool"]},{"id":"IL-0082","title":{"en":"Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse","ko":"EKS 노드 교체 중 CoreDNS 파드 하나가 종료되자 남은 CoreDNS 파드가 늘어난 부하를 버티지 못해 DNS 조회가 느려지고 타임아웃이 났다."},"org":"RebelMouse","org_slug":"rebelmouse","date":"2025-05-19","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0082","shared_components":["node-pool"]},{"id":"IL-0099","title":{"en":"User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes","ko":"메모리 상한 없이 8GB 만 보장한 프로필 때문에 워크숍 중 사용자 파드가 서로를 축출해 서버 시작이 15분을 넘고 커널이 죽었다."},"org":"2i2c","org_slug":"2i2c","date":"2025-05-12","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0099","shared_components":["node-pool"]},{"id":"IL-0027","title":{"en":"All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down","ko":"CoreDNS 파드가 모두 한 노드에 몰린 채 그 노드의 메모리가 고갈되자 DNS 가 응답하지 못해 아이디어스 서비스 전체가 멈췄다"},"org":"Backpackr","org_slug":"backpackr","date":"2025-05-07","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0027","shared_components":["node-pool"]},{"id":"IL-0007","title":{"en":"The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space","ko":"EKS 클러스터에서 VPC-CNI 가 파드마다 VPC IP 를 쓰다가 서브넷 IP 가 바닥나 파드가 IP 를 받지 못했다"},"org":"Adevinta","org_slug":"adevinta","date":"2024-10-22","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0007","shared_components":["subnet"]},{"id":"IL-0016","title":{"en":"Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady","ko":"네임스페이스와 규칙이 늘어 live 클러스터의 Prometheus 가 메모리 부족으로 OOMKilled 되며 플랫폼 모니터링이 끊겼다"},"org":"UK Ministry of Justice","org_slug":"uk-ministry-of-justice","date":"2023-07-25","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0016","shared_components":["node-pool"]},{"id":"IL-0070","title":{"en":"Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up","ko":"dask-gateway로 워커를 대량으로 띄우자 노드가 1분 만에 100대 늘었고, Prometheus가 메모리를 거의 다 쓰던 코어 노드에서 hub 파드를 비롯한 중요한 파드들이 축출되거나 재시작됐다"},"org":"2i2c","org_slug":"2i2c","date":"2023-02-01","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0070","shared_components":["node-pool"]},{"id":"IL-0094","title":{"en":"Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub","ko":"JupyterHub 클러스터에서 학생 파드가 노드의 CPU 와 메모리를 다 써서 자원 보장이 부족했던 코어 파드가 자원을 얻지 못했고, 허브 전체가 멈췄다."},"org":"2i2c","org_slug":"2i2c","date":"2020-10-28","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0094","shared_components":["node-pool"]},{"id":"IL-0013","title":{"en":"The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets","ko":"노드 RAM 을 절반으로 줄이자 conntrack 테이블이 가득 차 집계 서비스에 연결 리셋과 오류가 늘었다"},"org":"loveholidays","org_slug":"loveholidays","date":"2020-02-15","severity":"unknown","root_cause":"capacity","url":"https://benchlet.com/incidentlib/app/cases/IL-0013","shared_components":["node-pool"]}]