sequence parallelism
Ultra-Long Sequence Parallelism: Ulysses + Ring-Attention Principles and Implementation
00 — Preface Long-Sequence Training Challenges Training ultra-long sequences is a critical aspect of large model development. In real-world inference — especially within Agent pipelines — a model’s generalization ability to handle long contexts directly impacts its reliability. However, long-sequence scenarios place heavy demands on training resources due to the O(N²