HTTP/2

Go and ENHANCE_YOUR_CALM: Resolving HTTP/2 Interoperability Issues

Honghao Wang

— 3 min read
Go and ENHANCE_YOUR_CALM: Resolving HTTP/2 Interoperability Issues

Investigating the `ENHANCE_YOUR_CALM` Mystery in HTTP/2

In September 2025, a question appeared in our internal engineering chat:

> "Which part of our stack would be responsible for sending `ErrCode=ENHANCE_YOUR_CALM` to an HTTP/2 client?"

Two microservices were hitting a critical communication failure, and the team needed answers fast.

This post covers:

  • Background on HTTP/2 attack patterns that trigger Cloudflare defenses
  • A common Go library pitfall that can cause accidental PING floods
  • How we found and fixed the root cause
image

---

Understanding HTTP/2

HTTP/2 (RFC 9113) defines a binary wire format for HTTP semantics.

It wraps requests and responses into streams of:

  • HEADERS frames
  • DATA frames

Over TLS-secured TCP, HTTP/2 also sends control frames:

  • SETTINGS – advertise endpoint properties
  • WINDOW_UPDATE – manage flow-control
  • RST_STREAM – cancel a stream
  • GOAWAY – signal connection termination

Misuse Risks

The spec calls out denial-of-service considerations:

> "Implementations SHOULD track the use of these features and set limits on their use."

image

Cloudflare defenses include:

When malicious-like behavior is detected, Cloudflare closes the connection with GOAWAY and `ENHANCE_YOUR_CALM` (RFC link).

---

CVE-2019-9512 – The PING Flood

One common attack pattern:

> "The attacker sends continual pings, consuming CPU/memory resources."

PING frames (RFC §6.7) measure liveness and latency — but require work to acknowledge, leading to:

  • Potential exploitation via repeated pings
  • Mitigations that close the connection when ping rate is excessive

Go’s gRPC clients and Rust’s Hyper crate have historically triggered mitigations during adaptive window tuning.

---

Case Study: Microservices Mystery

We discovered two internal microservices were talking via Cloudflare edge, hitting the PING flood defense.

Why edge routing internally?

  • Dogfooding our infrastructure
  • Cloudflare Access secure auth
  • Easy Workers integration

Logs showed:

GODEBUG=http2debug=2

A minimal reproduction revealed:

RST_STREAM ... ErrCode=CANCEL
PING ...

Why RST + PING?

Found in grpc-io mailing list:

> "Sending a PING with an RST_STREAM lets clients distinguish unresponsive vs. slow servers."

But why so many resets?

---

Stream Closing Oddity

Logs:

.. received DATA END_STREAM len=0 ..
.. wrote RST_STREAM ..
.. wrote PING ..

According to the RFC stream state machine, END_STREAM should close the stream — RST_STREAM is unnecessary.

We noticed:

> RST+PING only happens when `resp.Body.Close()` is called without reading the body.

Even with an empty body, Go keeps the stream open until you consume the data.

---

The fix:

io.Copy(io.Discard, resp.Body) // Read entire body
resp.Body.Close()

Result: No RST_STREAM → no extra PING → no ENHANCE_YOUR_CALM connections closed.

---

Reading Bodies in Go – Gotcha

Problem: `json.Decoder` stops at the end of one JSON document, leaving data unread.

Better pattern:

defer func() {
    io.Copy(io.Discard, resp.Body)
    resp.Body.Close()
}()

This ensures:

  • HTTP/1.1 – connection reuse safety
  • HTTP/2 – avoids RST_STREAM + PING flood

---

Handling `ENHANCE_YOUR_CALM`

This error usually means:

  • Excessive concurrent streams
  • Frequent resets
  • Overuse of compression
  • Aggressive pinging
  • Capture packets (`SSLKEYLOGFILE` for TLS decryption)
  • Enable detailed trace logging (`GODEBUG=http2debug=2`)
  • Identify excessive feature use
  • Tune connection pooling/throttling

For Go:

  • Always read bodies fully — even empty ones — before close

---

Lessons Learned

  • Edge-routing internal calls exposes them to same defenses as external traffic
  • Reading bodies prevents subtle library behaviors from triggering defenses
  • Dogfooding surfaces real-world issues before customers hit them

---

image

Key takeaway: Subtle HTTP/2 misuse can cause production impact — precise debugging fixes both stability and performance.

Read more