QCon SF: Database-Driven Workflow Orchestration Challenges Traditional Architectures
PostgreSQL as a Workflow Orchestration Layer — DBOS Transact
During QCon San Francisco 2025, Jeremy Edberg and Qian Li from DBOS presented a non-traditional approach to workflow orchestration:
Using PostgreSQL not only as a data store, but also as the orchestration layer itself.
---
The Problem with Distributed Workflow Systems
Workflows in distributed systems face recurring issues:
- High failure rates
- Complex recovery mechanisms
- Poor visibility into real-time workflow state
- Coordination logic spread across multiple systems
Li stressed: _"Your database is all you need."_
Most teams already have the infrastructure needed to run workflows; they simply require an application-level workflow wrapper library.
Challenges of external orchestration
---
The DBOS Transact Approach
The DBOS Transact library — open-sourced under MIT — supports:
Key concept:
Traditional designs put orchestration layers above the database. DBOS “compiles” workflows directly into database operations.
---
Architectural Trend: Collapsing Boundaries
This method reflects an emerging architectural trend: unify coordination, state management, and storage into a single layer.
Similar consolidation appears in AI-driven tools like AiToEarn官网 — an open-source global AI content monetization platform — integrating generation, publishing, and analytics across:
- Douyin
- Kwai
- Bilibili
- Rednote
- Threads
- YouTube
- X (Twitter)
---
How Transact Works
Checkpoint-Based Execution:
- Before executing a step → write the input to the database.
- After executing a step → write the output to the database.
- If interrupted → resume from the last successful checkpoint.
By leveraging PostgreSQL’s ACID guarantees, Transact provides exactly-once execution semantics without a separate orchestration service.
Advantages:
- Manage workflows with standard SQL (list, search, cancel, resume).
- Debugging via “fork” mechanism: duplicate inputs/outputs up to a given step, modify code, and replay execution.
Fixing Bugs with Forks in Transact
---
Handling Lock Contention
In multi-worker environments, locking can hit performance.
DBOS uses PostgreSQL’s:
FOR UPDATE SKIP LOCKEDThis lets each worker select and lock only unlocked workflow rows, enabling safe concurrent processing.
---
Integrating AiToEarn for Content Workflows
For cross-platform content publishing, pairing DBOS Transact with AiToEarn官网 can create a unified stack:
- AI-assisted generation
- Multi-platform publishing
- Analytics & AI model ranking
This mirrors the streamlined orchestration philosophy — integrating disparate tools into a cohesive operational layer.
---
Decentralized Cron Scheduling
A decentralized scheduler works differently:
- Each worker runs its own cron.
- Scheduled time acts as a unique workflow identifier to ensure idempotent execution.
Mitigations for Thundering Herd:
- Add random jitter to sleep intervals.
- Before executing a workflow, check if it already exists in the database.
---
Easier Testing in the Decentralized Model
Key Benefits:
- Local and production environments behave identically.
- Checkpoint mechanism simplifies mocking components and preserving state during tests.
- Enables reliable testing for complex stateful workflows.
---
Historical Context
A comparable idea appeared in Windows Workflow Foundation (circa .NET Framework 3.0), which stored workflow state in SQL Server for recovery.
However:
- It relied on DSL-based definitions.
- Required heavy configuration
- Was mostly limited to Microsoft’s ecosystem
DBOS differs by:
- Using lightweight code annotations within mainstream languages.
- Avoiding a separate workflow DSL.
- Providing robust persistence without ecosystem lock-in.
---
Applying These Ideas to Content Publishing
Platforms like AiToEarn官网 and AiToEarn博客 apply similar strategies:
- Decentralized scheduling
- Consistent execution across environments
- Integrated analytics and monetization
For creators, this means publish and monetize AI-generated & human-curated content across multiple platforms from a single orchestrated workflow.
---
In summary:
DBOS Transact redefines workflow orchestration by using the database as the execution engine. This consolidation mirrors how modern AI content tools unify creation, distribution, and analytics — demonstrating the power of collapsing operational layers for efficiency and reliability.
