[{"id":"IL-0112","title":{"en":"The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing \"PLEG is not healthy\" NodeNotReady flaps and pod network timeouts","ko":"나중에 회수된 EKS 노드 AMI 로 노드를 교체한 뒤 노드가 PLEG is not healthy 를 보고하며 NotReady 를 오갔고 파드 네트워크 타임아웃이 났다"},"org":"DFDS","org_slug":"dfds","date":"2022-10-31","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0112","shared_components":["node-image","linux-kernel"]},{"id":"IL-0029","title":{"en":"Slow user-facing services from CPU throttling on containers that used far less CPU than their limit","ko":"CPU 한도를 건 컨테이너가 사용량이 한도에 한참 못 미치는데도 스로틀링되어 사용자 대면 서비스의 지연 시간이 늘었다."},"org":"Buffer","org_slug":"buffer","date":"2020-08-31","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0029","shared_components":["node-image","linux-kernel"]},{"id":"IL-0143","title":{"en":"GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image","ko":"GKE 1.13 과 1.14 의 특정 버전으로 업그레이드한 노드에서 커널 패닉이 늘었다."},"org":"Google Cloud","org_slug":"google-cloud","date":"2019-11-04","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0143","shared_components":["node-image","linux-kernel"]},{"id":"IL-0439","title":{"en":"Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cache","ko":"블록 디바이스와 파일 시스템을 확장할 때마다 데이터베이스 접근이 한동안 느려졌다"},"org":"Cybozu","org_slug":"cybozu","date":"2018-06-28","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0439","shared_components":["linux-kernel","database"]},{"id":"IL-0451","title":{"en":"Setting indexes INVISIBLE on Aurora MySQL 3.10 spiked CPU and ReadIOPS on all three readers but not on the writer","ko":"Aurora MySQL 에서 인덱스를 INVISIBLE 로 바꿀 때마다 Reader 3대의 버퍼풀이 무효화돼 CPU 와 ReadIOPS 가 튀었다"},"org":"Daangn","org_slug":"daangn","date":"2026-10-06","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0451","shared_components":["database"]},{"id":"IL-0313","title":{"en":"Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy pool","ko":"자체 운영 PgBouncer 로 옮기는 운영 카나리에서 일부 요청이 정확히 300초 멈추고 SQLAlchemy 커넥션 풀이 고갈됐다"},"org":"Decagon","org_slug":"decagon","date":"2026-08-05","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0313","shared_components":["database"]},{"id":"IL-0281","title":{"en":"PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 days","ko":"PostHog US 리전 Logs 제품의 ClickHouse 클러스터가 S3 에 둔 로그 데이터를 거의 다 지워 3일보다 오래된 로그가 사라졌다"},"org":"PostHog","org_slug":"posthog","date":"2026-02-19","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0281","shared_components":["database"]},{"id":"IL-0310","title":{"en":"An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactions","ko":"Aurora PostgreSQL 의 수동 장애 조치가 스스로 되돌아가 원래 writer 가 남고 백엔드의 쓰기 쿼리가 실패했다"},"org":"Hightouch","org_slug":"hightouch","date":"2025-10-23","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0310","shared_components":["database"]},{"id":"IL-0150","title":{"en":"A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1","ko":"DynamoDB DNS 자동화의 경쟁 조건이 us-east-1 리전 엔드포인트의 DNS 레코드를 비워 DynamoDB 와 EC2 인스턴스 시작이 실패했고, 그 여파로 EKS, ECS, Fargate 의 컨테이너 시작과 클러스터 스케일링이 14시간 넘게 막혔다"},"org":"Amazon Web Services","org_slug":"amazon-web-services","date":"2025-10-19","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0150","shared_components":["database"]},{"id":"IL-0171","title":{"en":"Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False","ko":"Bottlerocket AMI 에 libdcgm.so 가 없어 GPU 노드의 AcceleratedHardwareReady 가 False 가 됐고, Karpenter 노드 자동 복구가 Ready 뒤 10~11분마다 GPU 노드를 교체하는 일을 끝없이 되풀이했다."},"org":"Inflab","org_slug":"inflab","date":"2025-08-27","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0171","shared_components":["node-image"]},{"id":"IL-0436","title":{"en":"A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connections","ko":"Aurora MySQL 에서 INSERT 가 supremum 락에 막혀 커넥션이 고갈되고 freee 웹 서비스 전체가 다운되었다"},"org":"freee","org_slug":"freee","date":"2025-08-27","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0436","shared_components":["database"]},{"id":"IL-0147","title":{"en":"AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries","ko":"AKS 202508 노드 이미지가 캐시되지 않은 쿠버네티스 바이너리를 받을 때 HTTP 프록시 설정을 쓰지 않아, 프록시 뒤에 있는 노드의 프로비저닝이 CSE 종료 코드 31 로 실패했다."},"org":"Microsoft Azure","org_slug":"microsoft-azure","date":"2025-08-26","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0147","shared_components":["node-image"]},{"id":"IL-0247","title":{"en":"One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to Kubernetes","ko":"쿠버네티스로 옮긴 검색 시스템 Manas 에서 요청 백만 건에 한 건이 평소보다 100배 느려졌다"},"org":"Pinterest","org_slug":"pinterest","date":"2025-07-16","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0247","shared_components":["linux-kernel"]},{"id":"IL-0527","title":{"en":"OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc error","ko":"OpenStreetMap.org 의 복제 diff 생성이 invalid memory alloc request size 오류로 16시간 멈춤"},"org":"OpenStreetMap Foundation","org_slug":"openstreetmap-foundation","date":"2025-06-25","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0527","shared_components":["database"]},{"id":"IL-0329","title":{"en":"A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage","ko":"Postgres 17 업그레이드 뒤 다시 켠 PGAudit 가 마이그레이션 중 락을 쥔 채 멈춰, 서비스 전반이 느려지고 데이터베이스가 2분 동안 내려갔다"},"org":"incident.io","org_slug":"incident-io","date":"2025-04-09","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0329","shared_components":["database"]},{"id":"IL-0138","title":{"en":"A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days","ko":"nvidia-dkms-390 드라이버 빌드 실패로 망가진 커널 헤더가 몇 주 동안 방치됐다가, 정전 뒤 재부팅하자 하나뿐인 마스터 노드가 커널 패닉에 빠져 홈랩 쿠버네티스 클러스터가 6일 동안 멈췄다."},"org":"Barush Mendez","org_slug":"barush-mendez","date":"2025-03-03","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0138","shared_components":["linux-kernel"]},{"id":"IL-0347","title":{"en":"About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck","ko":"FR-PAR-1 가용 영역에서 하이퍼바이저 약 25% 의 VPC 연결이 11분 동안 끊겼다"},"org":"Scaleway","org_slug":"scaleway","date":"2024-10-18","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0347","shared_components":["database"]},{"id":"IL-0373","title":{"en":"Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four days","ko":"EU 리전의 Heroku Postgres Essential 등급 데이터베이스가 메모리가 바닥나 두 번에 걸쳐 연결을 받지 못했다"},"org":"Heroku","org_slug":"heroku","date":"2024-07-27","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0373","shared_components":["database"]},{"id":"IL-0024","title":{"en":"Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times","ko":"무거운 API 요청이 MongoDB 에 과부하를 걸고 취소되지 않은 요청이 쌓여 Hugging Face Hub 가 사흘 동안 세 번 멈췄다"},"org":"Hugging Face","org_slug":"hugging-face","date":"2024-04-22","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0024","shared_components":["database"]},{"id":"IL-0217","title":{"en":"Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage","ko":"스키마 캐시가 데워지지 않아 주 MySQL 데이터베이스가 내부 교착에 빠지고 Honeycomb 전체가 68분 동안 멈췄다"},"org":"Honeycomb","org_slug":"honeycomb","date":"2023-07-25","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0217","shared_components":["database"]},{"id":"IL-0330","title":{"en":"Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 seconds","ko":"Go database/sql 커넥션 풀을 기다리던 요청이 2주 동안 간헐적으로 타임아웃됐고, 한 요청은 거의 20초를 기다렸다"},"org":"incident.io","org_slug":"incident-io","date":"2023-04-20","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0330","shared_components":["database"]},{"id":"IL-0385","title":{"en":"GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recovering","ko":"큰 MySQL 테이블의 스키마 마이그레이션 마지막 단계에서 읽기 복제본 다수가 semaphore deadlock 에 걸려 GitHub 핵심 서비스가 2시간 50분 저하됐다"},"org":"GitHub","org_slug":"github","date":"2021-11-27","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0385","shared_components":["database"]},{"id":"IL-0146","title":{"en":"New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNI","ko":"Central India 에서 새로 만든 Azure CNI 클러스터의 VM 이 높은 값의 MAC 주소를 받으면서 bridge 모드 Azure CNI 의 오래된 버그가 드러나 터널 연결이 끊기고 kubectl exec 와 파드 간 통신이 타임아웃됐다."},"org":"Microsoft Azure","org_slug":"microsoft-azure","date":"2020-12-04","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0146","shared_components":["node-image"]},{"id":"IL-0196","title":{"en":"Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data","ko":"Redis Operator 로 레플리카를 늘리면서 메모리 한도를 낮추자 데이터를 가진 유일한 Redis 파드가 재시작됐고, Sentinel 이 빈 레플리카를 마스터로 올려 데이터가 모두 지워졌다."},"org":"Palark","org_slug":"palark","date":"2020-03-13","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0196","shared_components":["database"]},{"id":"IL-0012","title":{"en":"CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts","ko":"CPU limit 과 커널 CFS 쿼터 버그로 파드가 과하게 스로틀링돼 헬스 체크가 실패하고 애플리케이션이 멈췄다"},"org":"Omio","org_slug":"omio","date":"2020-02-17","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0012","shared_components":["linux-kernel"]},{"id":"IL-0486","title":{"en":"Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailable","ko":"Figma의 PostgreSQL 데이터베이스에서 쓰기 IOPS와 락 경합이 치솟아 서비스가 하루 동안 여러 번 내려갔다"},"org":"Figma","org_slug":"figma","date":"2020-01-22","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0486","shared_components":["database"]},{"id":"IL-0241","title":{"en":"A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day","ko":"데이터베이스 shard 하나가 primary 를 뽑지 못해 Stripe API 요청 대부분이 두 차례 실패했다"},"org":"Stripe","org_slug":"stripe","date":"2019-07-10","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0241","shared_components":["database"]},{"id":"IL-0277","title":{"en":"Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachable","ko":"Elastic Cloud us-east-1 의 ZooKeeper 조정 계층이 쿼럼을 잃어 Elasticsearch 배포에 6시간 넘게 접속하지 못하고 Kibana 는 18:44 UTC 까지 불안정했다"},"org":"Elastic","org_slug":"elastic","date":"2019-02-04","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0277","shared_components":["linux-kernel"]},{"id":"IL-0388","title":{"en":"A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guilds","ko":"Redis 고가용 클러스터의 primary 노드가 내려간 뒤 Discord API 가 느려지고 길드 일부가 오프라인으로 보이며 접속이 실패했다"},"org":"Discord","org_slug":"discord","date":"2017-10-13","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0388","shared_components":["database"]},{"id":"IL-0454","title":{"en":"Baemin billing servers could not open new MariaDB connections at the start of a card-discount event","ko":"배민 카드사 할인 이벤트가 시작되자 빌링 시스템이 MariaDB 에 새 커넥션을 맺지 못해 결제 요청 일부가 실패했다"},"org":"Woowa Brothers","org_slug":"woowa-brothers","date":"2016-12-08","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0454","shared_components":["database"]},{"id":"IL-0536","title":{"en":"Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster","ko":"US East 의 Amazon SimpleDB 가 두 시간 동안 거의 모든 API 호출에 실패했다"},"org":"Amazon Web Services","org_slug":"amazon-web-services","date":"2014-06-13","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0536","shared_components":["database"]},{"id":"IL-0485","title":{"en":"Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slow","ko":"계획된 OS 업그레이드 작업이 운영 중인 데이터베이스 서버를 재설치해 Dropbox 서비스가 멈췄다"},"org":"Dropbox","org_slug":"dropbox","date":"2014-01-10","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0485","shared_components":["database"]},{"id":"IL-0589","title":{"en":"Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings table","ko":"Kickstarter 의 모든 MySQL 복제본이 중복 키 오류로 복제를 멈췄다"},"org":"Kickstarter","org_slug":"kickstarter","date":"2013-03-07","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0589","shared_components":["database"]},{"id":"IL-0303","title":{"en":"GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down","ko":"GitHub 의 새 3노드 MySQL 클러스터에서 자동 페일오버가 잘못 일어나고 클러스터 상태가 갈라져 GitHub.com 이 두 번 멈췄다"},"org":"GitHub","org_slug":"github","date":"2012-09-10","severity":"unknown","root_cause":"software-bug","url":"https://benchlet.com/incidentlib/app/cases/IL-0303","shared_components":["database"]}]