Migrate from harp-python#
harp-data is the successor to harp-python for reading Harp binary data files into pandas DataFrames. The core concepts of device schemas, register maps and binary files are unchanged, but the API has been reorganized to separate data reading from device communication.
This guide covers the three workflows most users relied on in harp-python.
Installation#
harp-python was a single distribution. Reading data now needs only harp-data, since serial transport and the device client are separate packages.
Before
After
For the full toolkit of serial transport, device client and data reading, install the metapackage instead:
Load a device schema at runtime#
In harp-python, harp.create_reader() accepted a dataset folder and handled schema loading internally. open_dataset is the direct replacement, and it finds the device.yml inside the folder automatically.
Before
After
If the schema lives outside the data folder, pass it explicitly:
List the session contents#
reader.contents maps the name of every register with data in the folder to its address, in address order, which is the quickest way to see what was recorded:
reader = open_dataset("session.harp")
print(reader.contents) # {'WhoAmI': 0, 'DigitalInputState': 32, ...}
The reader also holds the compiled device module at reader.device_module, so a register class can be accessed without keeping a separate variable:
Note
The old harp.read_schema() had no direct equivalent that needed calling separately. open_dataset handles schema loading in one step, matching the convenience of the original API.
Read a single register#
The old API allowed reads through attribute access on the reader. The new API inverts this, so reader.read() takes the register class, its name, or its address as the argument.
Before
# by attribute name
df = reader.AnalogData.read()
# by name string
df = reader.registers["AnalogData"].read()
# by address
df = reader.registers[44].read()
After
# by register class (accessed through the reader)
df = reader.read(reader.device_module.AnalogData)
# by name, the direct analogue of the old string lookup
df = reader.read("AnalogData")
# by address
df = reader.read(44)
Names resolve against the device address space rather than the module namespace, so access to a core register such as reader.read("WhoAmI") works even though a device module does not name it.
Absolute timestamps#
The epoch parameter keeps its name and moves from create_reader to open_dataset. harp-python also allowed it per read. harp-data sets it once for the dataset, so every register is read on the same clock.
Before
reader = harp.create_reader("session.harp", epoch=harp.REFERENCE_EPOCH)
df = reader.AnalogData.read()
After
from harp.data import REFERENCE_EPOCH
reader = open_dataset("session.harp", epoch=REFERENCE_EPOCH)
df = reader.read(reader.device_module.AnalogData)
Read the whole session at once#
There is no read_all(). Whole-session loading is a comprehension over contents, which keeps the choice of what to load with the caller:
Bitmask registers lose their per-flag columns by default#
This is the change most likely to break working code. harp-python always expanded a bitmask register into one boolean column per flag, so a script could select a flag by name. harp-data returns a single integer column instead, and expands the flags only when asked.
Before
After
Group masks need no such flag. harp-python mapped each value to its member name, and harp-data decodes them by default, as a pd.Categorical rather than plain strings, so a comparison against a string still works.
Parameter reference#
| harp-python | harp-data | Notes |
|---|---|---|
keep_type=True |
keep_type=True |
Same, but the column is named message_type rather than MessageType |
epoch=REFERENCE_EPOCH |
epoch=REFERENCE_EPOCH |
Same, but set on open_dataset rather than per read |
epoch=None |
epoch=None (default) |
Float seconds, same |
| inferred from the first frame | time_index=False |
Needed for data carrying no timestamp |
| always on | demux_bit_masks=True |
Needed to keep per-flag columns |
| always on | decode_enums=True (default) |
Group mask values, now pd.Categorical |
Schemaless read#
For a raw .bin file with no schema, or a quick look at the data, the read() function works the same as before. Only the import path changes.
Before
After
from harp.data import read
df = read("Behavior_44.bin")
df = read("Behavior_44.bin", keep_type=True)
Both functions infer the payload type and element count from the frame, so no register metadata is needed. The new one assumes timestamped data, which is what a device sends, and takes time_index=False for the rare buffer that is not. It also takes epoch per call, since a single file has no dataset to set one on.
Static device packages#
Loading a YAML at runtime is convenient. However, a generated device package gives the same interface without parsing a schema at startup, and its registers resolve under IDE autocompletion and a type checker.
It also validates what it opens. The package declares its own WHO_AM_I, which the reader checks against the device.yml in the folder. A session recorded from a different device then fails on construction rather than decoding the files against the wrong register map. A module built from that same folder cannot catch this, since it agrees with the folder by definition.
Such a package is an ordinary Python module under the harp.device namespace. Import it, pass it to open_dataset, and the rest of the API is identical:
from harp.device import behavior
from harp.data import open_dataset
reader = open_dataset("session.harp", behavior)
# a register class now resolves statically
df = reader.read(behavior.AnalogData)
A generated module starts up faster and resolves under a type checker, which a module built from a schema at runtime cannot. See Registers from a schema for how device modules are structured.