Vortex

Vortex is a columnar file format designed to support both efficient random access and efficient compressed storage. Like Parquet, it integrates into the Arrow data ecosystem, which makes it worth considering if your existing data preparation tools work well with Arrow tables. Unlike litData and MDS, Vortex files are self-describing and do not ship with a separate index.json file to describe how their contents are structured and stored. You can think of Vortex as a modern Parquet with fast random access. You can read more about the Vortex format’s structure in the project’s documentation.

Preparing Vortex shards. To use the Vortex format with Zephon, install the project with the vortex extra which includes the vortex-data dependency (note that Vortex requires Python >= 3.11):

examples/guide/datasets/vortex_install.sh

# Install Zephon with the vortex extra (needs Python 3.11 or newer).

pip install "zephon[vortex]"

Let’s look at an example of writing a small collection of records into Vortex files:

examples/guide/datasets/vortex_write_shards.py

"""Write records into Vortex shards."""

import vortex as vx

records = [{"text": f"sample {i}", "label": i % 2} for i in range(1000)]

# One file per shard; 64-256 MB each is the range we recommend.
for shard, start in enumerate(range(0, len(records), 250)):
    array = vx.array(records[start : start + 250])
    vx.io.write(array, f"data/vortex_demo/shard{shard}.vortex")

The Vortex python library can also write data from Arrow tables, which provides a path for converting existing tabular datasets into this format:

examples/guide/datasets/vortex_from_arrow.py

"""Convert an Arrow table into a Vortex shard."""

import pyarrow as pa
import vortex as vx

table = pa.table({"text": ["first", "second"], "label": [0, 1]})

vx.io.write(vx.array(table), "data/vortex_demo/shard0.vortex")

Although Vortex does not require a separate index file, Zephon provides an indexing tool that collects and packages the shard names, file sizes, and record counts into an index.json file that helps for efficiently planning the work to be done during training without requiring Zephon to load each Vortex shard individually:

examples/guide/datasets/vortex_index.sh

# Build an index.json so Zephon need not open every shard to plan.

python -m zephon.build_index vortex data/vortex_demo

Reading Vortex shards in Zephon. Once the Vortex files have been written, you can define a Dataset object by passing the dataset location and the fmt="vortex" argument to the Dataset.from_path function. Zephon will infer the format if fmt is not passed. You can then use the DatasetInspector to examine the decoded sample payloads:

examples/guide/datasets/vortex_dataset.py

"""Define and inspect a Vortex Dataset."""

from zephon.debug import DatasetInspector
from zephon.io import Dataset

# Vortex files are self-describing, so the index is an optimization, not a
# requirement. Compression is handled inside the format: no extra local copy.
dataset = Dataset.from_path("vortex_demo", "data/vortex_demo", fmt="vortex")

with DatasetInspector(dataset) as inspector:
    print(inspector.read(shard_id=0, sample_index=0))

Note that Vortex handles compression operations within the file format itself, so Zephon reads the .vortex files directly without requiring an extra, decompressed copy of the shard to be created.

Vortex structs return dictionaries keyed by field name, including single-field structs. Files containing bare values, such as strings or numeric lists, return those values directly. Both use the same conversion rules as Parquet.

Caching reads. Zephon caches Vortex metadata and encoded file segments in memory to speed up repeated reads. Each fetch replica has its own caches, separate from the on-disk shard cache. You can adjust their sizes through the pipeline’s io_options; these are the defaults:

pipeline.options(
    io_options={
        "metadata_cache_entries": 256,
        "vortex": {"segment_cache_bytes": "64mb"},
    }
)

Set either limit to zero to disable that cache. The metadata limit counts footers; the segment size is a memory target that Vortex can exceed. Decoded samples and other memory use are additional, so allow some headroom when planning memory usage.

Worker threads. Vortex uses a background thread pool shared by readers in the same process. Zephon defaults to one background worker to avoid multiplying thread counts across fetch processes and training ranks. Fetch threads can still read concurrently. You can change the pool size through the pipeline’s io_options:

pipeline.options(io_options={"vortex": {"worker_threads": 1}})