SplitDataset

Julia package for reading datasets split blockwise into multiple HDF5 files. It is the Julia analogue of SplitDataset in Python.

Provides a H5SplitDataset that can be treated as a DiskArray

To load files, use the folder-argument constructor.

H5SplitDataset(folder::String) # opens a SplitDataset from a folder containing h5 and json files

To write whole arrays, use the following constructor.

H5SplitDataset(
    folder,
    a::AbstractArray{T,N},
    block_size::NTuple{N,Int},
)

Writing not-block-aligned parts of arrays is currently not supported. This package is an intermediate solution until Zarr supports all the features we need.