Skip to content

harp-data#

Load Harp register data into pandas DataFrames. This is the package that pulls in pandas. harp-protocol stays numpy-only and exposes a pandas-free ColumnData view that this package assembles into a DataFrame.

There are two ways in, depending on what is on disk:

  • a whole dataset folder holding many registers, read with DatasetReader
  • a single register file or buffer, read with parse_to_dataframe

Read a whole dataset folder#

A Harp acquisition is usually saved as a de-multiplexed folder, one binary file per register, named <DeviceName>_<address>.bin, alongside the device.yml schema for the device:

📦 session.harp
 ┣ 📜 Behavior_0.bin
 ┣ 📜 Behavior_44.bin
 ┣ ...
 ┗ 📜 device.yml

Reading is based on a device module that describes how to decode each register. open_dataset supplies one automatically. It finds the device.yml in the folder, builds the module, and returns a ready-to-use reader:

from harp import data

reader = data.open_dataset("session.harp")
df = reader.read("AnalogData")  # by name
df = reader.read(44)            # by address

contents maps every register with data in the folder to its address, keyed by register name. It is the place to start on an unfamiliar dataset, and since its keys are exactly what read takes, loading a whole dataset can be done with a comprehension:

reader.contents  # {'WhoAmI': 0, 'AnalogData': 33, ...}

frames = {name: reader.read(name) for name in reader.contents}

A name is resolved through the device register map rather than the module namespace, so the core registers are accessible by name too.

The schema describes the structure regardless of what was recorded. This means a register declared in the device register map with no data present in the folder reads as an empty DataFrame carrying the same columns. contents is what distinguishes the two cases. A register the device does not declare at all raises KeyError.

Given a device module already in hand, either a pre-generated package or one built with create_device_module, pass it as the second argument:

from harp import data
from harp.device import behavior

reader = data.open_dataset("session.harp", behavior)
df = reader.read(behavior.AnalogData)  # by register class

Prefer the register class where a generated package supplies one, since it is the only form that type-checks and a misspelling is caught before the folder is read. A module built by create_device_module resolves its registers as Any, so there the class verifies no more than the name does.

The Harp time becomes the DataFrame index named "Time", as float seconds by default or an absolute DatetimeIndex when the dataset is opened with epoch=REFERENCE_EPOCH. The anchor is set once for the dataset, since it describes how the recording was made rather than how one register is read. Data carrying no timestamp raise unless time_index=False is passed. Multi-chunk registers logged as <DeviceName>_<address>_<suffix>.bin are concatenated in filename order. Pass a resolver to support an alternative on-disk layout. paths reports what the resolver found, keyed by address, which is where a custom layout or a chunked register can be checked.

The <DeviceName> prefix comes from the DEVICE_NAME declared by the device module. Pass name= to override it, or to supply one when the module declares an empty name.

When a device module declaring an identity is supplied and the folder carries a device.yml, their whoAmI values are checked against each other. Reusing a module across sessions and opening the wrong folder then fails on construction rather than decoding the files against the wrong register map. Pass validate=False to turn off every check the reader performs, so a folder whose device.yml is damaged can be read with a module obtained elsewhere.

Read a single register file#

parse_to_dataframe takes a register and a source, either a path, bytes, or an open binary file, and returns one row per frame:

from harp import data
from my_device import AnalogData

df = data.parse_to_dataframe(AnalogData, "AnalogData.bin")
df = data.parse_to_dataframe(
    AnalogData, raw, time_index=True, epoch=None, keep_type=False, decode_enums=True
)

time_index decides the index: True, the default, gives the Harp time named "Time", and False gives a RangeIndex. epoch anchors that index, giving float seconds when omitted and an absolute DatetimeIndex when set to a datetime such as REFERENCE_EPOCH. This function reads one file rather than a dataset, so it takes the anchor directly. Enum fields decode to pd.Categorical, and decode_enums=False keeps raw codes.

From an already-parsed payload#

Given a batched payload already in hand, for example from register.parse_bulk, convert it directly:

from harp import data

_data, timestamps, _msg, payload = AnalogData.parse_bulk(raw)
df = data.payload_to_dataframe(payload)

Write data back out#

to_file and to_buffer are the inverse of the readers, encoding values as Harp frames. Useful for round-tripping data or generating test corpora:

from harp import data

data.to_file(AnalogData, values, "AnalogData.bin", timestamps=seconds)

harp-data is released as open source under the MIT license. Bug reports and contributions are welcome at the GitHub repository.

harp.data.open_dataset(root, device_module=None, *, schema=None, name=None, resolver=default_file_resolver, converters=None, require_converters=True, epoch=None, validate=True) #

open_dataset(
    root: str | PathLike[str],
    device_module: M,
    *,
    name: str | None = ...,
    resolver: FileNameResolver = ...,
    epoch: datetime | None = ...,
    validate: bool = ...,
) -> DatasetReader[M]

open_dataset(
    root: str | PathLike[str],
    device_module: None = ...,
    *,
    schema: str | PathLike[str] | None = ...,
    name: str | None = ...,
    resolver: FileNameResolver = ...,
    converters: Mapping[str, Any] | None = ...,
    require_converters: bool = ...,
    epoch: datetime | None = ...,
    validate: bool = ...,
) -> DatasetReader[DeviceModule]

Open a de-multiplexed Harp dataset folder and return a :class:DatasetReader.

If the device module is omitted, the schema file inside the folder will be used. The device.yml inside root is first built into a module using :func:~harp.device.schema.create_device_module.

If a device module is provided, its identity class will be used to validate the dataaset, and a generated package additionally carries register classes a type checker can verify.

schema points at the schema file when it isn't root/device.yml, and converters and require_converters are forwarded to :func:~harp.device.schema.create_device_module for custom interfaceType decoding. These three parameters describe alternative ways to supply a module, so they are mutually exclusive, and will raise when more than one is specified.

epoch anchors the time index of every read to absolute time, since the anchor describes the recording rather than one register. The reference Harp clock starts at :data:REFERENCE_EPOCH, and the default of None gives float seconds.

validate cannot rescue a corrupt device.yml if that schema file is also used to build the module. Reading such a folder always requires supplying a module obtained elsewhere.

harp.data.read(source, *, time_index=True, epoch=None, keep_type=False) #

Read the binary data of a single register, inferring its native layout.

source may be a file path, raw bytes, or an open binary file. The element type, length and timestamp presence are read from the first frame; values decode to the matching native numpy type (no enum or bit-mask decoding). The remaining options match :func:~harp.data.parse_to_dataframe.

harp.data.DatasetReader #

Bases: Generic[M]

Reader over a de-multiplexed Harp dataset folder.

Construct from a device module and a dataset folder, then read the frames of a register into a DataFrame by register class, by name, or by address::

reader = DatasetReader(behavior, "session.harp")
df = reader.read(behavior.AnalogData)  # by register class
df = reader.read("AnalogData")         # by name
df = reader.read(44)                   # by address

:func:open_dataset builds one for a folder that carries its own device.yml. :attr:contents lists what was recorded, keyed by register name.

device_module is a device module -- a generated device package, or one built from a schema with :func:~harp.device.schema.create_device_module. Its REGISTER_MAP is read at construction.

The files are matched by a <DeviceName> prefix, taken from the DEVICE_NAME declared by the module. Pass name to override it, or to supply one when the module declares an empty name.

When the folder carries a device.yml and the module declares an identity, their whoAmI values are checked against each other. A module paired with the wrong folder then fails here rather than decoding the files against the wrong register map. validate turns off every check the reader performs, so a folder whose device.yml is damaged can be read with a module obtained elsewhere.

The reader is typed on the module it was given, so registers stay accessible through :attr:device_module at whatever precision that module offers.

File resolution defaults to the Harp file format: <name>_<address>.bin and, when a register was logged as several <name>_<address>_<suffix>.bin chunks, they are concatenated in filename order. Pass resolver (a :data:FileResolver) to support an alternative on-disk layout.

epoch anchors the time index of every read to absolute time, so one dataset is read on one clock rather than the choice being made per register. It describes how the recording was anchored. The reference Harp clock starts at :data:REFERENCE_EPOCH.

contents property #

The mapping from register name to address for registers with data under :attr:root.

device_module property #

The device module this reader parses against, as the type it was given.

A generated package resolves each register to its own class; one built by :func:~harp.device.schema.create_device_module resolves them collectively, the same ceiling as accessing it directly.

name property #

The <DeviceName> prefix used to match binary files.

paths property #

The mapping from address to binary files discovered under :attr:root.

root property #

The dataset folder being read.

read(register, *, suffix=None, time_index=True, keep_type=False, decode_enums=True, demux_bit_masks=False) #

Read the data of one register into a DataFrame.

register is a register class, a register name, or an address. Names are resolved through the device register map rather than the module namespace, which declares no core registers. Prefer the class where a generated package supplies one, since it is the only form a type checker can verify. A module built by :func:~harp.device.schema.create_device_module resolves its registers as Any, so there the generated module verifies no more than the name does.

A register declared in the device register map with no data present in the folder reads as an empty DataFrame carrying the same columns, since the schema describes the structure of the data regardless of whether anything was recorded. :attr:contents is what tells the two cases apart. A register the device does not declare at all raises KeyError.

suffix selects a single <name>_<address>_<suffix>.bin chunk, and naming one that is absent raises FileNotFoundError (default: concatenate every chunk for the address). The remaining options match :func:~harp.data.parse_to_dataframe, except that the epoch is the one the reader was opened with.

harp.data.default_file_resolver(root, name) #

Harp file format resolver: map address -> sorted <name>_<address>... files.

harp.data.parse_to_dataframe(register, source, *, time_index=True, epoch=None, keep_type=False, decode_enums=True, demux_bit_masks=False) #

Parse all frames of register from source into a DataFrame.

source may be a file path, raw bytes, or an open binary file object. time_index makes the Harp time the DataFrame index, named "Time", and False leaves a RangeIndex. epoch anchors that index to absolute time, giving a DatetimeIndex measured from it, where the default of None gives float seconds; the Harp clock starts at :data:REFERENCE_EPOCH. keep_type inserts a leading column; decode_enums controls whether enum fields become pd.Categorical (True) or raw codes; demux_bit_masks expands each flag (BitMask) field into one boolean column per flag member (True) or keeps it as a single raw-integer column.

harp.data.payload_to_dataframe(payload, *, decode_enums=True, demux_bit_masks=False, copy=False) #

Turn a (batched) payload into a DataFrame, one row per frame.

decode_enums relabels enum columns as pd.Categorical; demux_bit_masks expands each flag (BitMask) column into one boolean column per flag member.

harp.data.to_file(register, values, file, *, timestamps=None, message_type=MessageType.Event, port=255) #

Write values as register frames to file (see :func:to_buffer).

harp.data.to_buffer(register, values, *, timestamps=None, message_type=MessageType.Event, port=255) #

Encode values as a flat buffer of register frames.

values is a payload (scalar or batch) or an ndarray of the payload_class.payload_dtype; timestamps (length-N seconds) makes every frame timestamped; message_type is one :class:MessageType or a length-N array (e.g. the msgtype view from parse_bulk).

harp.data.REFERENCE_EPOCH = datetime(1904, 1, 1) module-attribute #

Harp reference epoch, time zero of the Harp clock in UTC.