Skip to content

Move Blackrock nsx handling closer to the buffer API pattern#1820

Open
h-mayorquin wants to merge 2 commits intoNeuralEnsemble:masterfrom
h-mayorquin:add_buffer_api_like_behavior_for_blackrock
Open

Move Blackrock nsx handling closer to the buffer API pattern#1820
h-mayorquin wants to merge 2 commits intoNeuralEnsemble:masterfrom
h-mayorquin:add_buffer_api_like_behavior_for_blackrock

Conversation

@h-mayorquin
Copy link
Contributor

PR #1513 introduced BaseRawWithBufferApiIO, a base class that stores lightweight buffer descriptions (file paths, offsets, dtypes) instead of persistent np.memmap objects, creating short-lived mmap views on demand through cached file descriptors. This PR moves BlackrockRawIO toward that same pattern. We cannot fully adopt BaseRawWithBufferApiIO because: (a) Blackrock's standard format (v2.2/2.3/3.0) stores multiple data blocks with explicit headers in a single file, requiring block-aware offset tracking that the base class's one-buffer-per-stream model does not support; (b) the PTP variant interleaves per-sample timestamps with sample data in fixed-size packets, requiring strided mmap access (np.ndarray with custom strides) rather than contiguous views; and (c) the segmentation logic for PTP files needs to read timestamps from the file to detect gaps, which couples parsing and segmentation in a way the base class does not allow. However, we can adopt the core idea: replace stored memmaps with on-demand creation.

This has the added benefit of reducing the number of memmaps that are open at any time and we have seen that this pattern is both 1) more performant memory wise, 2) less likely to lead to leaks. Importantly, this PR does not change any behavior of the reader, it is an internal optimization only. It lays the groundwork for a follow-up PR adding gap detection on the standard format.

@h-mayorquin h-mayorquin self-assigned this Feb 17, 2026
@h-mayorquin h-mayorquin changed the title Blackrock nsx closer to the buffer API pattern Move Blackrock nsx handling closer to the buffer API pattern Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant