Xie Saining, Fei-Fei Li, and Yann LeCun Team Up for the First Time! Introducing the New "Hyperception" Paradigm — AI Can Now Predict and Remember, Not Just See
Spatial Intelligence & Supersensing: The Next Frontier in AI
Leading AI researchers — Fei-Fei Li, Saining Xie, and Yann LeCun — have been highlighting a transformative concept: Spatial Intelligence.
This goes beyond simply “understanding images or videos.” It’s about:
- Comprehending spatial structures
- Remembering events
- Predicting future outcomes
In essence, a truly capable AI should not only “see,” but also sense, understand, and actively organize experience — a core competency for future multimodal intelligence.
---
Introducing Cambrian-S: Spatial Supersensing in Video
The trio recently collaborated to publish Cambrian-S: Towards Spatial Supersensing in Video.
Their proposed paradigm — Supersensing — emphasizes that AI models must:
- Observe, recognize, and answer
- Remember and understand 3D structure
- Predict future events
- Organize experiences into an Internal World Model
Co-first author Shusheng Yang noted: supersensing demands active prediction, filtering, and organization of sensory inputs — not passive reception.
Saining Xie described Cambrian-S as their first step into spatial supersensing for video. Despite its length, the paper is rich with detail and groundbreaking perspectives — essential reading for those in video-based multimodal modeling.
---
Defining “Spatial Supersensing”
Background
Last year, Xie’s team released Cambrian-1 — focusing on multimodal image modeling.
Instead of rushing towards Cambrian-2 or Cambrian-3, they paused to reflect:
- What truly is multimodal intelligence?
- Is LLM-style perception modeling adequate?
- Why is human perception intuitive yet powerful?
Their conclusion: Without Supersensing, there can be no Superintelligence.
---
What Supersensing Is — and Isn’t
Supersensing is not about better sensors or cameras. It’s how a digital lifeform truly experiences the world — continuously absorbing sensory streams and learning from them.
Key point: Human agents can solve abstract problems without perception, but to operate in the real world, AI agents require sensory modeling.
As Andrej Karpathy has remarked: sensory modeling may be all that intelligence requires.
---
The Five Levels of Supersensing
- No Sensory Capabilities
- Example: LLMs
- Operate purely on text/symbols, without grounding in the physical world.
- Semantic Perception
- Parsing pixels into objects, attributes, and relationships.
- Strong performance in “image captioning.”
- Streaming Event Cognition
- Handling continuous, real-time data streams.
- Active interpretation and response to ongoing events.
- Implicit 3D Spatial Cognition
- Understanding video as a projection of the 3D world.
- Answering “what exists,” “where it is,” and “how it changes over time.”
- Current video models are still lacking here.
- Predictive World Modeling
- Anticipating possible future states based on prior expectations.
- Using surprise as a driver for attention, memory, and learning.
---
Why Predictive World Modeling Matters
The human brain continuously predicts potential world states. High prediction errors trigger attention and learning, but current multimodal systems lack such mechanisms.
---
Evaluating Spatial Supersensing
Existing video benchmarks focus on language understanding and semantic perception, but neglect higher-level supersensing.
Even efforts like VSI-Bench address spatial perception only for short videos, missing unbounded, continuous visual streams.
---
VSI-SUPER: A New Benchmark
VSI-SUPER introduces two challenging tasks:
- VSI-SUPER Recall (VSR)
- Recall anomalous object locations in long videos.
- VSI-SUPER Count (VSC)
- Continuously accumulate spatial information across long scenarios.
Unlike conventional tests, VSI-SUPER stitches short clips into arbitrarily long videos, challenging models to remember unbounded visual streams.
Testing Gemini-2.5 Flash revealed high performance on standard benchmarks, yet failure on VSI-SUPER.
---
Cambrian-S: Addressing the Challenge
While data and scale matter, the key missing factor is training data designed for spatial cognition.
VSI-590K Dataset
- 590,000 samples from:
- First-person indoor environments (3D annotations)
- Simulators
- Pseudo-labeled YouTube videos
Models from 0.5B to 7B parameters were trained, achieving up to +30% spatial reasoning gains over baseline MLLMs.
Still, VSI-SUPER remains unfixed — suggesting LLM-style multimodal systems are not the ultimate path to supersensing.
---
Predictive Perception Prototype
Cambrian-S features:
- Strong general video/image understanding
- Leading spatial perception performance
Additional strengths:
- Generalizes to unseen spatial tasks
- Passes debias stress tests
---
Human Analogy
In baseball, players predict ball trajectory before their brain fully processes the visual input — a skill rooted in internal predictive world models.
This ability filters sensory overload by ignoring low-error predictions & focusing on surprises.
---
Latent Frame Prediction
The team trained a latent frame prediction (LFP) head that:
- Predicts the next input frame
- Compares predicted vs actual (difference = surprise value)
- Uses surprise for:
- Memory management (skip unremarkable frames)
- Event segmentation (detect scene boundaries)
Even a smaller model using this method beat Gemini on VSI-SUPER.
---
Related Releases
- Multimodal Benchmark Design research — removing language bias
- Simulator tools for collecting spatial perception video data
---
Open-Source Resources
- Project homepage: https://cambrian-mllm.github.io/
- Paper: https://arxiv.org/abs/2511.04670
- Code: https://github.com/cambrian-mllm/cambrian-s
- Models: https://huggingface.co/collections/nyu-visionx/cambrian-s-models
- Dataset: https://huggingface.co/datasets/nyu-visionx/VSI-590K
- Benchmark: https://huggingface.co/collections/nyu-visionx/vsi-super
---
Connection to AI Content Platforms
Advanced cognitive capabilities like predictive perception could power AI-driven creators.
Platforms such as AiToEarn官网 allow:
- AI content generation
- Multi-platform publishing (Douyin, Kwai, WeChat, Bilibili, Rednote, YouTube, LinkedIn, Instagram, Pinterest, Threads, X)
- Integrated analytics & model ranking
Read AiToEarn Blog | Open-source Repo
---
Summary:
Spatial Supersensing represents a paradigm shift in AI development — moving from passive perception to active, predictive, and structurally aware intelligence. Benchmarks like VSI-SUPER and models like Cambrian-S are critical steps toward this future, where AI can remember, predict, and organize experiences across unbounded multimodal streams.
