h5torch package

class h5torch.file.File(path: str, mode: Literal['r', 'r+', 'x', 'w-', 'a', 'w'] = 'r', driver=None)

Initializes a file handle to a HDF5 file.

Parameters:

  • path (str) – path to HDF5 file to save (or read) to (or from).

  • mode (Literal["r", "r+", "x", "w-", "a", "w"], optional) – load in the file in read, write, append, …, mode, by default “r”.

  • driver (Optional[str]) – see https://docs.h5py.org/en/stable/high/file.html#file-drivers. By default None. Choose “core” to load file in an in-memory format.

register(data: List | ndarray | Tuple[ndarray, ndarray, Sequence], axis: int | Literal['central', 'unstructured'], length: int | None = None, name: str | None = None, mode: Literal['N-D', 'csr', 'coo', 'vlen', 'separate'] = 'N-D', dtype_save: str | None = None, dtype_load: str | None = None, csr_load_sparse: bool = False) → None

Registers a new dataset to a HDF5 file.

Parameters:

  • data (Union[List, np.ndarray, Tuple[np.ndarray, np.ndarray, Sequence]]) –

    Data to save.

    If mode == “N-D” or “csr”, expects an np.ndarray

    If mode == “coo”, expects either a 2D np.ndarray or a Tuple: indices (N, M), values (M) and shape (..)*N

    If mode == “vlen”, expects a List of 1D np.ndarrays

    If mode == “separate”, expects a List of np.ndarrays

  • axis (Union[int, Literal["central", "unstructured"]]) – Axis to align the dataset to. The first dataset that should be registered to a HDF5 file should always be the central dataset.

  • length (Optional[int], optional) – length of the dataset, useful when registering a dataset to which you want to append later by default None, which means you will not be able to append to the dataset later.

  • name (Optional[str], optional) – name of the dataset, ignored in the case of axis == “central”, mandatory in the case of an alignment axis, by default None

  • mode (Literal["N-D", "csr", "coo", "vlen", "separate"], optional) – mode in which to save the data, by default “N-D”

  • dtype_save (Optional[str], optional) – data type in which to save the data, by default None, which means it uses the datatype as given

  • dtype_load (Optional[str], optional) – data type in which to load the data, by default None, which means it uses the datatype as given

  • csr_load_sparse (bool, optional) – For csr objects, whether it should be loaded as a dense array or a sparse collection of indices and data.

append(data: str, name: str) → None

Append data to an existing group.

Parameters:

  • data (str) – The data to append

  • name (str) – The key / name of the HDF5 dataset to append data to.

class h5torch.dataset.Dataset(file: str | File | h5pyDict, sampling: int | Literal['coo'] = 0, subset: Tuple[str, str] | ndarray | None = None, sample_processor: Callable | None = None, in_memory: bool = False)

h5torch.Dataset object.

Parameters:

  • file (Union[str, h5torch.File, h5torch.file.h5pyDict],) – Either string: Path to the saved HDF5 file. Underlying file has to follow the logic defined by h5torch.File. Or: a h5torch.File handle opened in read mode. Or: a h5pyDict, the type of dict created by calling .todict() on a h5torch.File.

  • sampling (Union[int, Literal["coo"]], optional) – Sampling axis, by default 0

  • subset (Optional[Union[Tuple[str, str], np.ndarray]], optional) – subset of data to use in dataset. Either: a np.ndarray of indices or np.ndarray containing booleans. Or: a tuple of 2 strings with the first specifying a key in the dataset and the second a regex that must match in that dataset. By default None, specifying to use the whole dataset as is.

  • sample_processor (Optional[Callable], optional) – A callable that takes as input arguments f (the file handle to the HDF5 file) and sample (the output of this Dataset’s __getitem__). Can be used to postprocess samples By default None

  • in_memory (bool, optional) – Whether to load the h5torch dataset in memory completely. Allows for speed ups in data-loading. By default False. Redundant if a h5pyDict is passed, as everything is already in memory.

class h5torch.dataset.SliceDataset(file: str | File | h5pyDict, sampling: int | Literal['coo'] = 0, sample_processor: Callable | None = None, window_size: int = 501, overlap: int = 0, window_indices: ndarray | None = None, in_memory: bool = False)

h5torch.SliceDataset object. Takes slices from the central object (and the sampled axis) as samples. The default behavior is to take slices starting from the first element with size window_size and optionally overlapping by overlap elements. If the last slice of the data would be an incomplete sample, it would be thrown away.

If window_indices is specified, then window_size and overlap is ignored.

Parameters:

  • file (Union[str, h5torch.File, h5torch.file.h5pyDict],) – Either string: Path to the saved HDF5 file. Underlying file has to follow the logic defined by h5torch.File. Or: a h5torch.File handle opened in read mode. Or: a h5pyDict, the type of dict created by calling .todict() on a h5torch.File.

  • sampling (Union[int, Literal["coo"]], optional) – Sampling axis, by default 0

  • sample_processor (Optional[Callable], optional) – A callable that takes as input arguments f (the file handle to the HDF5 file) and sample (the output of this Dataset’s __getitem__). Can be used to postprocess samples By default None

  • window_size (int, optional) – Size of the slices in number of elements, by default 501

  • overlap (int, optional) – Overlap of each slice in number of elements, by default 0

  • window_indices (Optional[np.ndarray], optional) – A np.ndarray of size N x 2 with N the number of slices. Each row specifies the start and end index of each slice. (End indices are not included in python-slicing style) Can be used to overwrite window_size and overlap default behavior and/or to specify subsets as training/validation/test sets. By default None

  • in_memory (bool, optional) – Whether to load the h5torch dataset in memory completely. Allows for speed ups in data-loading. By default False.