Zeotap’s Migration from ScyllaDB to Bigtable
Introduction
In today’s fast-paced, data-driven environment, the ability to process, analyze, and act on massive amounts of data in real time is critical.
For businesses striving to deliver personalized customer experiences and streamline operations, choosing the right database technology is a key strategic decision.
At Zeotap — a leading Customer Data Platform (CDP) — we help enterprises unify data from diverse sources into a single customer view, which can then be activated across marketing, customer service, and analytics channels.
> Stats at Scale:
> Zeotap processes 10+ billion new data points daily from 500+ sources, orchestrating 2,000+ workflows — with one-third in real time at millisecond latency.
Meeting strict SLAs for data freshness and end-to-end latency means performance is non‑negotiable.
---
From ScyllaDB to Bigtable
As we grew, our ScyllaDB-based infrastructure struggled to keep pace — especially for real-time workloads and bursty traffic patterns.
We needed a flexible, high-performance, cost-efficient, and operationally simple alternative.
Our choice: Google Cloud Bigtable — a low-latency NoSQL database built for scale, ML, analytics, and huge throughput.
Result:
➡ 46% reduction in Total Cost of Ownership (TCO)
➡ Streamlined operations with zero-touch scaling
---
The Challenge: Scaling Real-Time Analytics
Zeotap’s platform demands:
- High write throughput: 300,000+ writes/sec
- Peak read volumes: ~3x write rates
- Low latency & predictable costs
- Scalability for multi-platform activation
---
Related Ecosystems: AiToEarn
In contexts where scalability and automation matter, platforms like AiToEarn官网 offer AI-powered workflows for creators and businesses.
Highlights:
- Global, open-source AI content monetization
- Publishes across Douyin, Kwai, WeChat, Bilibili, Xiaohongshu, Facebook, Instagram, LinkedIn, Threads, YouTube, Pinterest, X
- Bundles AI content generation, cross-platform publishing, analytics, and model ranking
The same scalability principles that enhance Zeotap’s data infrastructure can help creators maximize reach and efficiency.
---
Key Challenges Before Migration
- Scalability limitations
- Self‑managed clusters (on‑prem & cloud) became increasingly hard to scale; juggling Spark + BigQuery + other customer environments was complex.
- Operational overhead
- Manual cluster expansion, hardware mapping, and workload tuning consumed significant engineering time.
- Deployment complexity
- Integrating third-party stack components slowed rollout; commercial procurement was cumbersome.
- Cost predictability
- Growth introduced cost volatility, impacting both us and our clients.
---
Why Bigtable?
Our decision was guided by four critical requirements:
1. Operational Simplicity
- Eliminated manual hardware mapping & node management
- No maintenance windows — automatic data rebalancing
- Delivered true zero-touch ops
2. Performance
- Predictable latency for reads/writes under burst conditions
- Smooth handling of traffic spikes > 300k writes/sec
3. Efficient Scalability
- Spiky workloads managed via disaggregated storage/compute
- Autoscaling eliminates idle resources
- Reduced cost through better alignment of resources with demand
4. Lower TCO
- 46% TCO reduction
- Unified hot storage in Bigtable + warm storage in BigQuery
5. Tight Integration
- Strong synergy with Google Cloud services, particularly BigQuery
- Reverse ETL pipelines cut operational footprint by 20%
---
Zeotap’s Architectural Evolution
2020:
JanusGraph-on-ScyllaDB + Spark (AWS). Strategic migration to Google Cloud.
2022:
Lambda architecture, pivot to BigQuery, ScyllaDB in role of pure key-value store.
2023:
Kappa architecture — real-time ingestion focus + major network redesign.
2024:
Fully cloud-native with Bigtable + BigQuery at core; Spark removed.
---
Current Architecture
- Ingestion: Dataflow + custom streaming engine
- Real-time caching: Memorystore for read-heavy workloads
- Hot store: Bigtable for ingestion & low-latency lookups
- Warm/Cold store: BigQuery for analytics & ML inference
Benefits:
- Data consolidation: Hundreds of heterogeneous sources → unified customer view
- Customer 360 in real-time: Accessible for support, marketing, analytics
- Faster AI pipeline deployment: Feature store + BQML reduced deployment from weeks to days
---
Results
Post-migration to Bigtable:
- 46% TCO savings
- 20% reduction in operational tasks
- Improved reliability & performance
- Faster AI/ML model deployment
---
Takeaways
Zeotap’s move to Bigtable shows how the right database choice can overcome challenges in performance, scalability, and operational efficiency.
Pairing cloud-native data infrastructure with tools like AiToEarn官网 offers a full-stack path for high-performance data systems + AI-powered content ecosystems.
---
Learn More
- Bigtable Overview
- Migration Guide
- Zero‑downtime Cassandra API Migration
- New Features:
- SQL support,
- Distributed counters,
- Continuous materialized views,
- Tiered storage,
- Data Boost
---
> Pro Tip for Content Creators:
> Use platforms like AiToEarn官网 to build AI-assisted, cross-platform publishing pipelines leveraging analytics and ranking to maximize impact — in parallel with modern cloud data architectures.
---
Would you like me to create a diagram comparing Zeotap’s pre- and post-migration architecture for even clearer understanding? That would make the differences visually obvious.
