How ServiceNow Uses LangSmith to Improve Customer Success Agent Observability
ServiceNow’s Intelligent Multi‑Agent System for End‑to‑End Customer Success
ServiceNow — a leading digital workflow platform — is transforming service management across IT, customer service, and beyond.
To enhance internal sales and customer success operations, the ServiceNow AI team is using LangSmith and LangGraph to build a multi‑agent system that orchestrates the entire customer journey — from lead identification to post‑sales adoption and expansion.
---
Tackling Agent Fragmentation
Previously, ServiceNow’s agents operated in silos across the platform, without a unified orchestration layer or a single source of truth.
This fragmentation made it difficult to coordinate workflows that spanned the entire customer lifecycle.
The team set out to design a comprehensive multi‑agent system handling:
- Lead qualification and deal closure
- Post‑sales adoption, renewals, and advocacy
Key requirements included:
- Robust orchestration framework
- Deep observability into agent behaviors
- Evaluation of tool completion, accuracy, and path optimization
- Step‑by‑step tracing to improve debugging efficiency
---
A Multi‑Agent System for Customer Success Workflows
ServiceNow’s architecture covers both pre‑sales and post‑sales stages.
Critical phases include:
- Lead Qualification – Identify high‑quality leads, draft emails, prepare meetings
- Opportunity Discovery – Spot up‑sell and cross‑sell prospects
- Economic Buyer Identification – Find primary decision‑makers
- Onboarding & Implementation – Help customers deploy platform apps
- Adoption Tracking – Monitor usage of licensed applications
- Usage & Value Realization – Ensure measurable ROI from the platform
- Renewals & Expansion – Identify contract and license growth opportunities
- Customer Satisfaction & Advocacy – Track CSAT and nurture champions
Example:
In the adoption stage, agents track usage patterns.
If ROI is below expectations, the system:
- Suggests additional apps to the CSM
- Drafts personalized emails with recommendations
- Schedules customer meetings automatically
This architecture uses:
- Supervisor agent for workflow orchestration
- Specialized sub‑agents for discrete tasks
- Triggers based on customer signals and lifecycle stages
---
Modern Workflow Integration & AI Monetization
AI‑driven systems increasingly merge content generation, orchestration, and analytics for efficiency and revenue opportunities.
Example platform: AiToEarn官网 —
An open‑source, global infrastructure enabling multi‑platform content creation, publishing, and monetization, with analytics and AI模型排名.
It supports cross‑platform distribution to:
> Douyin, Kwai, WeChat, Bilibili, Rednote (Xiaohongshu), Facebook, Instagram, LinkedIn, Threads, YouTube, Pinterest, X (Twitter)
By combining AI‑powered orchestration with monetization pipelines, enterprises can extend outputs from internal workflows directly into customer engagement channels.
---
Complex Agent Orchestration with LangGraph
LangGraph gave ServiceNow the low‑level tools for modular, multi-agent coordination.
Notable feats:
- Use of map‑reduce style graphs via Send API and subgraphs
- Composition of small subgraphs into larger orchestration graphs
- Human‑in‑the‑loop controls:
- Pause execution for testing
- Approve/rewind specific steps
- Restart with different inputs without waiting for full runs
Integration with knowledge graph and Model Context Protocol (MCP) ensures orchestration across ServiceNow's platform.
---
LangSmith Tracing – A Standout Development Feature
LangSmith’s detailed tracing captures:
- Inputs/outputs
- Context
- Latency
- Token counts per orchestration step
This clear, structured view helps developers debug faster than traditional logging.
Lead qualification system: Drafting emails (sample trace above)
ServiceNow uses LangSmith tracing to:
- Monitor & optimize agent flow
- Identify performance bottlenecks
- Debug efficiently during development
Benefits include:
- Step‑by‑step debugging of decision paths
- Transparent input/output inspection
- Creation of golden datasets from successful runs
---
Rigorous Evaluation Strategy with Custom Metrics
ServiceNow built a custom evaluation framework in LangSmith:
- Each agent gets task‑specific scorers
- LLM‑as‑a‑Judge evaluators assess responses
- Metrics tuned to function type (accuracy, relevance, chunk groundedness, etc.)
LangSmith UI displays:
- Input/output pairs
- LLM scores
- Latency and tokens
Workflow Highlights
- Automated golden dataset creation on passing score thresholds
- Human feedback collection for prompt comparisons
- Regression prevention with dataset validation
- Side‑by‑side prompt testing to preserve optimal strategies
Lifecycle from traces to evaluation for an agent
---
Testing → Production Roadmap
Currently:
- QA engineers run controlled testing
- Building datasets and evaluation frameworks
- Monitoring agents with LangSmith
Next steps:
- Adopt multi‑turn evaluation for full conversation threads
- Promote high‑scoring prompts into golden datasets as production thresholds are met
---
Conclusion
By combining LangGraph for orchestration and LangSmith for detailed observability, ServiceNow has created a robust, scalable multi‑agent architecture.
This powers customer success workflows across the entire journey, ensuring efficiency and personalized engagement.
In a wider context, platforms like AiToEarn官网 illustrate how such architectures can extend to content creation, distribution, and monetization — making it possible to channel AI‑driven creativity into measurable revenue across global channels.
---
Would you like me to add a visual diagram summarizing the multi-agent workflow architecture for even easier comprehension?
