Cloudflare Epic Outage: A Database Permission Change Crippled Half the Internet

Cloudflare Epic Outage: A Database Permission Change Crippled Half the Internet
# Cloudflare's November 18, 2025 Global Outage: A Detailed Analysis

On the evening of **November 18, 2025 (Beijing time)**, global network infrastructure provider **Cloudflare** experienced a **major outage**.

This incident disrupted many prominent internet services worldwide, including **ChatGPT**, **X (Twitter)**, and **Spotify**.

Industry observers noted:  
> “This event absolutely deserves its own chapter in the history of internet disasters.”

![image](https://blog.aitoearn.ai/content/images/2025/11/img_001-490.jpg)

---

## Overview of the Incident

### Key Facts

- **Occurrence Time:**  
  November 18, 2025, **19:20–22:30** (primary impact window)
- **Duration:**  
  Approximately **3 hours** for core outage; about **6 hours** to full restoration
- **Scope:**  
  **Global**
- **Root Cause:**  
  Not a cyberattack. A latent bug caused by **internal database permission changes** led to abnormal ballooning of bot management configuration files, triggering software crashes across Cloudflare’s worldwide nodes.

---

## Event Timeline

- **19:05:** Change deployed to ClickHouse database access control  
- **19:28:** Change took effect — failures begin  
- **19:32–21:05:** Investigation underway  
- **21:05:** First-stage mitigation applied (core issue persists)  
- **21:37:** Root cause identified  
- **22:24:** Abnormal config generation halted; nodes rolled back to stable files  
- **22:30:** Core services restored  
- **01:06 (next day):** Full recovery

![image](https://blog.aitoearn.ai/content/images/2025/11/img_002-458.jpg)

---

## Impact Scope

This outage cascaded across the internet, affecting nearly half of global online services:

- **AI Platforms:** ChatGPT, Claude, Perplexity  
- **Social Services:** X (Twitter), Spotify, Discord, Grindr  
- **Gaming Platforms:** League of Legends, Minecraft servers  

**Symptoms:**  
- “500 Internal Server Error” messages  
- Continuous human verification prompts

---

## Official Explanation

Cloudflare’s detailed incident report can be found here:  
[https://blog.cloudflare.com/18-november-2025-outage/](https://blog.cloudflare.com/18-november-2025-outage/)

![image](https://blog.aitoearn.ai/content/images/2025/11/img_003-432.jpg)

### Technical Summary

- The outage stemmed from **a permissions change in the database system**.  
- This triggered excessive entries in the bot management **feature configuration file**, doubling its size beyond acceptable software limits.  
- Oversized files propagated globally, causing routing software failures.

---

## How Cloudflare Operates

Cloudflare’s CDN powers **~20% of world websites**, delivering content from the nearest data center with latencies under **50ms** to 95% of the global population.

**Benefits include:**
- Faster site performance  
- Reduced server load for operators  
- Security against malicious bots and attacks  

On outage night, **CTO Dane Knecht** stressed it was *not an attack*, but a vulnerability triggered by routine changes.

---

### Root Technical Cause

1. **Bot Management Feature File**:  
   Updated every few minutes to defend against threats  
   - Input: ClickHouse query results
2. **Permission change**  
   - Allowed retrieval of metadata from additional databases  
   - Duplicated feature rows added unexpectedly  
3. **File size doubled**, exceeding routing software limits  
4. Core proxy systems crashed, returning HTTP 5xx errors  
5. Workers KV and Access services also impacted  

Initially misdiagnosed as a **DDoS attack**, engineers later identified the oversized file as the true cause.

---

## Remediation Measures

Cloudflare plans to:
- Enforce **stricter file size limits**  
- Validate auto-generated configuration files more robustly  
- Use **staged rollouts** with continuous monitoring

---

## Business Implications for Creators

Cloudflare’s outage highlights the risk of **single points of failure** in internet infrastructure. Platforms like [AiToEarn官网](https://aitoearn.ai/) — an open-source global AI content monetization system — can help creators:
- Publish across major channels (Douyin, Kwai, WeChat, Bilibili, Rednote, Facebook, Instagram, LinkedIn, Threads, YouTube, Pinterest, X)  
- Integrate AI content creation, analytics, and model ranking ([AI模型排名](https://rank.aitoearn.ai))  
- Maintain visibility and revenue during network disruptions

---

## Cloudflare’s Statement

> This incident was our most serious failure since 2019.  
> Outages of this magnitude are unacceptable.  
> Our architecture is designed for high fault tolerance; after every failure, we improve resilience.

**Compensation:**  
- No plan announced yet  
- **Business** and **Enterprise** customers covered by SLA for sub-99.9% availability  
- Possible **10–20% account credit** for ~4.5 hours downtime

**Additional:**  
X (Twitter) is collecting compensation claims for paid users affected.

![image](https://blog.aitoearn.ai/content/images/2025/11/img_004-406.jpg)

---

## Discussion

Were **you** impacted by Cloudflare’s outage? Share your experience and thoughts below.

---

## References

1. [Last night, Cloudflare's global outage took down half the internet!](https://mp.weixin.qq.com/s/XmM9pjejZcMfH3gtO5DyZg?scene=1&click_id=30)  
2. [Cloudflare’s worst outage in six years: one line of Rust code halted half the world’s traffic](https://mp.weixin.qq.com/s/Lx2BiBiQPgsA5gbpJlNl3Q?scene=1)  
3. [Cloudflare blog: November 18, 2025 outage](https://blog.cloudflare.com/18-november-2025-outage/)

---

*Source compilation: Kuozhanmi AIGC, infoQ, dbaplus community*

Read more

Translate the following blog post title into English, concise and natural. Return plain text only without quotes. 哈佛大学 R 编程课程介绍

Harvard CS50: Introduction to Programming with R Harvard University offers exceptional beginner-friendly computer science courses. We’re excited to announce the release of Harvard CS50’s Introduction to Programming in R, a powerful language widely used for statistical computing, data science, and graphics. This course was developed by Carter Zenke.