Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode Skip to content
Zephon
Light Logo Dark Logo

User guide

  • Basic concepts
  • Why Zephon?
  • Working with datasets
    • JSONL
    • Parquet
    • Binary shard formats
      • litData
      • Mosaic (MDS)
      • Vortex
      • Lance
    • Storage backends and shard cache
  • WorkSources: the training curriculum
    • Mixing datasets
    • Shuffling and repeating samples
  • Pipelines: from a curriculum to training batches
    • How pipelines run
    • Preparing samples
    • Shuffling and maintaining mixtures
    • Packing
    • Building training batches
    • Inspecting and tuning pipelines
    • Distributed training
    • Checkpointing and resuming pipelines
    • User-defined operators
  • Training integrations

Internals

  • Internals overview
  • WorkSources and work chunks
  • Elastic determinism
  • Operators and runners
  • Records and sample lifecycles
  • Checkpoint replay and bounded recovery

API reference

  • Datasets
  • WorkSources
  • Pipeline
  • Operators
  • Records and batches
  • Runtime and tuning
  • Tools
Back to top
Copyright © 2026, DatologyAI
Made with Sphinx and @pradyunsg's Furo