[{"id":"IL-0114","title":{"en":"An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours","ko":"클러스터 용량을 늘리던 중 버전이 고정되지 않은 VPC CNI 플러그인이 자동으로 업데이트돼 새 파드가 IP 를 받지 못하고 Pending 에 머물렀고, 메인넷 운영 환경의 정산이 약 3.5시간 멈췄다."},"org":"CoW Protocol","org_slug":"cow-protocol","date":"2024-04-19","severity":"unknown","root_cause":"config-change","url":"https://benchlet.com/incidentlib/app/cases/IL-0114","shared_components":["node-pool","vpc-cni"]},{"id":"IL-0019","title":{"en":"An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device","ko":"EKS 모듈 18 업그레이드로 새 노드의 디스크가 20Gb 로 줄어 이미지 풀이 no space left on device 로 실패했다"},"org":"UK Ministry of Justice","org_slug":"uk-ministry-of-justice","date":"2023-09-18","severity":"unknown","root_cause":"config-change","url":"https://benchlet.com/incidentlib/app/cases/IL-0019","shared_components":["node-pool","vpc-cni"]},{"id":"IL-0091","title":{"en":"Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together","ko":"Terraform 으로 노드 인스턴스 타입을 바꾸자 노드가 한꺼번에 교체되어 사용자 운영 파드가 모두 지워지고 서비스가 내려갔다"},"org":"UK Ministry of Justice","org_slug":"uk-ministry-of-justice","date":"2023-06-06","severity":"unknown","root_cause":"config-change","url":"https://benchlet.com/incidentlib/app/cases/IL-0091","shared_components":["node-pool"]},{"id":"IL-0109","title":{"en":"A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes","ko":"로그 수집을 고치려던 EKS 노드 롤오버가 꼬이면서 Auto Scaling 그룹이 정상 워커 노드를 모두 종료해 클러스터에 노드가 하나도 남지 않았다."},"org":"DFDS","org_slug":"dfds","date":"2023-02-14","severity":"sev1","root_cause":"config-change","url":"https://benchlet.com/incidentlib/app/cases/IL-0109","shared_components":["node-pool"]},{"id":"IL-0105","title":{"en":"Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503","ko":"VPC CNI 교체로 ALB 에서 타깃 그룹이 떨어져 나가 쿠버네티스 클러스터의 모든 서비스가 503 을 냈다"},"org":"DFDS","org_slug":"dfds","date":"2021-06-22","severity":"unknown","root_cause":"config-change","url":"https://benchlet.com/incidentlib/app/cases/IL-0105","shared_components":["vpc-cni"]},{"id":"IL-0055","title":{"en":"A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster","ko":"실패하는 CronJob 이 기본 concurrencyPolicy 때문에 겹겹이 쌓여 재시작하는 파드가 수천 개로 늘었고 클러스터가 내려갔다"},"org":"Target","org_slug":"target","date":"2018-09-20","severity":"unknown","root_cause":"config-change","url":"https://benchlet.com/incidentlib/app/cases/IL-0055","shared_components":["node-pool"]}]