Content-addressed FlatBuffers serialization for distributed data access in the SPIDER experiment

The SPIDER experiment, located in Padua at the Neutral Beam Test Facility (NBTF), has the objectives of improving ion source performance, optimizing the use of cesium vapor, and verifying the uniformity and stability of the extracted ion beam during long-duration pulses. Achieving approximately 100 KV significantly increases the complexity of the neutral beam system, making it one of the most advanced auxiliary heating systems ever developed for a fusion reactor. Currently, the diagnostic and control system of SPIDER relies on a well-established data acquisition and storage framework known as MDSplus. This system efficiently handles both fast transient data acquisition and long-pulse data streaming. However, despite its robustness and proven performance, MDSplus's fixed communication protocol, which is typically one of its key strengths, can become a limiting factor when multiple heterogeneous clients concurrently request data access. This issue is particularly evident when several users attempt to retrieve live signal updates from the server during an ongoing experimental session, while data are still in the online state and has not yet been transferred to offline storage. To overcome this limitation, this work presents an implementation of a flexible communication protocol layer that enhances data distribution capabilities of MDSplus. Unlike traditional centralized storage, distributed systems can handle growing data volumes more efficiently by distributing data across multiple servers (nodes) in a connected network. By combining the efficient ordered search and disk-friendliness of the B-Tree with the cryptographic verifiability and content-addressing of the Merkle Tree, Prolly Trees (probabilistic B-tree) are a good fit for distributed systems. In particular, the use of content-defined chunking to guarantee structural determinism makes them naturally compatible with InterPlanetary Linked Data's (IPLD) content-addressed storage model. Nodes are stored as immutable IPLD blocks keyed by their hashes, allowing peers to detect shared subtrees, avoid redundant transfers, and validate received data through hash recomputation. The implementation of the Polly Tree over IPLD leverages the use of FlatBuffers to reduce the overhead given by the IPLD serialization of nodes. A middleware is responsible for collecting data produced by a device and creating the node to store. At the same time, this middleware will enable a publish/subscribe mechanism for a real-time stream with the processed nodes among multiple clients, avoiding the redundancy of asking to the storage the same content.