Airbnb Engineering

Airbnb’s Mussel V2: Next-Generation Key-Value Store Unifying Streaming and Batch Ingestion

Honghao Wang

— 2 min read

Mussel v2 — Airbnb’s Next-Generation Key-Value Store

Airbnb's engineering team has released Mussel v2, a complete rearchitecture of its internal key-value engine.

The new design unifies streaming and bulk ingestion, simplifies operations, and scales to handle significantly larger workloads.

Key Performance Highlights

  • > 100,000 streaming writes/sec sustained
  • Handles tables over 100 TB
  • p99 read latency under 25 ms
  • Bulk workflows ingest tens of terabytes

This enables product teams to focus on innovation without dealing with complex data pipeline management.

---

Limitations of Mussel v1

The earlier Mussel v1:

  • Powered Airbnb's internal data services
  • Used static hash-partition design on Amazon EC2
  • Managed via Chef scripts
  • Separate batch and streaming ingestion paths increased operational overhead
  • Consistency enforcement was more complex
Mussel v1 Architecture

Mussel V1 architecture (Source: Airbnb Engineering Blog)

---

Mussel v2: Modern Architecture

Mussel v2 integrates:

  • NewSQL backend
  • Kubernetes native control plane
  • Elasticity of object storage
  • Low-latency caching
  • Operability of modern service meshes

Core Improvements

  • Kubernetes manifests with automated rollouts
  • Dynamic range sharding + presplitting to avoid hotspots
  • Namespace-level quotas and cost transparency dashboards

---

Dispatcher Layer

  • Stateless and horizontally scalable
  • Routes client API calls
  • Manages retries
  • Supports dual write & shadow read modes for migration

---

Durable Writes via Kafka

  • Writes persisted into Kafka
  • Replayer and Write Dispatcher apply writes to backend in order
  • Bulk loads via Airflow jobs + S3 staging
  • Merge/replace semantics preserved

---

Expiration Service

  • Topology-aware expiration
  • Shards data namespaces into range subtasks
  • Parallel deletion by multiple workers
  • Minimizes live query impact
  • Targeted deletes for write-heavy tables
Mussel v2 Architecture

Mussel V2 architecture (Source: Airbnb Engineering Blog)

---

Migration from v1 to v2

Migrating from v1’s eventually consistent backend to v2’s strongly consistent architecture was complex. Airbnb’s team followed a blue-green deployment strategy:

Migration Steps

  • Table-level granularity deployment
  • Continuous validation + fallback mechanisms
  • Bootstrap tables using backups & sampled data for presplitting
  • Verify checksums after ingestion
  • Replay lagging Kafka events
  • Enable dual writes
  • Gradually shift reads to v2
  • Shadow traffic monitors consistency
  • Fallback to v1 on error spikes
Migration Pipeline

Data migration pipeline from Mussel V1 to V2 (Source: Airbnb Engineering Blog)

---

Operational Challenges

  • Write deduplication
  • Controlled retries
  • Adjusting query execution and workload distribution
  • Per-table staging + automated fallback
  • Monitoring enabled migration of > 1 PB with zero downtime

---

Broader Ecosystem Note

Platforms like AiToEarn官网 offer similar unification in AI content workflows:

  • Open-source, global AI content monetization
  • AI-driven generation, cross-platform publishing, analytics, and model ranking
  • Supports major platforms: Douyin, Kwai, WeChat, Bilibili, Rednote, Facebook, Instagram, LinkedIn, Threads, YouTube, Pinterest, X (Twitter)
  • Reduces operational friction for creative and technical communication

Explore:

---

Summary:

Mussel v2 demonstrates the power of modern orchestration + scalable backend architecture. Its migration and operational improvements highlight best practices for large-scale, high-consistency workloads — lessons equally applicable to AI content and multi-platform automation environments.

---

Do you want me to create a side-by-side comparison table for Mussel v1 vs Mussel v2 so the differences are clearer? That would make this rewrite even more digestible.

Read more