SDK Reference

IOWarp SDK: Interprocess Communication, Runtime Modules, Context Transfer, Assimilation

Section 1: Hermes Shared Memory (Interprocess Communication)

High-performance shared memory for multi-process IPC. Hermes provides lock-free data structures and zero-copy communication, enabling efficient data sharing between processes without serialization overhead.

Key APIs

  • • hermes::ipc::MemoryBackend
  • • hermes::ipc::Allocator
  • • hermes::data_structures::*

Data Structures

  • • hipc::string — Shared memory string
  • • hipc::vector<T> — Shared memory vector
  • • hipc::list<T> — Shared memory list
  • • hipc::unordered_map<K,V> — Shared memory map

Memory Backends

POSIX SHM

Standard POSIX shared memory for Linux/Unix systems

mmap

Memory-mapped files for persistent shared memory

CUDA

GPU HBM (High Bandwidth Memory) for GPU-accelerated workloads

Code Example

#include <hermes/hermes_shm.h>

// Get the global memory manager
auto mem = HERMES_MEMORY_MANAGER;

// Create an allocator for POSIX SHM
auto alloc = mem->GetAllocator<PosixShmMmap>(alloc_id);

// Create a shared memory vector
auto vec = hipc::make_mptr<hipc::vector<int>>(alloc);
vec->push_back(42);
vec->push_back(100);

// Access from another process using the same alloc_id
// The vector is automatically synchronized across processes

Performance Characteristics

  • • Zero-copy: Data structures live directly in shared memory, no serialization
  • • Lock-free: Atomic operations for thread-safe access
  • • Type-safe: C++ templates ensure compile-time type checking

Section 2: Chimaera Runtime Modules

Extensible task-based runtime system. Modules register as shared libraries loaded dynamically, enabling plugin-based architecture for IOWarp components.

Key Concepts

  • Module: A shared library (.so) that implements the Chimaera module interface
  • Task: A unit of work submitted to a module for execution
  • TaskState: Lifecycle state (Pending, Running, Completed, Failed)
  • Worker Threads: Thread pool that executes tasks asynchronously

Lifecycle

  1. Create module — Load shared library and initialize
  2. Submit tasks — Queue work items for processing
  3. Process results — Retrieve completed tasks and handle outcomes

Code Example

#include 

// Create a module instance
chimaera::ModuleManager manager;
auto module = manager.LoadModule("libwrp_cte_core.so");

// Create a task
chimaera::Task task;
task.SetFunction("ProcessData");
task.SetArgs({data_ptr, data_size});

// Submit task
auto task_id = module->SubmitTask(task);

// Wait for completion
auto result = module->WaitForTask(task_id);

// Process result
if (result.GetState() == chimaera::TaskState::Completed) {
    auto output = result.GetOutput();
    // Use output...
}

Module Interface

All Chimaera modules must implement the following interface: 

class ChimaeraModule {
public:
    virtual void Initialize(const Config& config) = 0;
    virtual TaskId SubmitTask(const Task& task) = 0;
    virtual TaskResult WaitForTask(TaskId id) = 0;
    virtual void Shutdown() = 0;
};

Section 3: CLIO Transfer SDK

Hierarchical data movement across storage tiers. CLIO Transfer automatically manages data placement based on access patterns, cost, and performance requirements.

Key APIs

wrp::cte::Put()

Store data in a bucket with automatic tier placement

wrp::cte::Get()

Retrieve data from a bucket, promoting to faster tiers if needed

wrp::cte::PartialPut()

Store a portion of data (offset + size)

wrp::cte::PartialGet()

Retrieve a portion of data (offset + size)

Code Example

#include <wrp_cte/wrp_cte.h>

// Create a bucket (logical container for related data)
wrp::cte::Bucket bkt("my_data");

// Put data into the bucket
float* temperature_data = /* ... */;
size_t data_size = 1024 * sizeof(float);
bkt.Put("temperature", temperature_data, data_size);

// Get data from the bucket
float* output = new float[1024];
bkt.Get("temperature", output, data_size);

// Partial operations
bkt.PartialPut("temperature", temperature_data + 512, 
               512 * sizeof(float), 512 * sizeof(float));
bkt.PartialGet("temperature", output + 512, 
               512 * sizeof(float), 512 * sizeof(float));

Blob Operations

CLIO Transfer treats all data as blobs (binary large objects) with optional metadata:

  • • Automatic compression
  • • Checksum verification
  • • Versioning support

Bucket Management

Buckets provide logical organization and access control:

  • • Hierarchical naming (e.g., simulation/temperature)
  • • Metadata queries
  • • Access policies

Context-Aware Placement

CLIO Transfer uses machine learning models to predict data access patterns and automatically move data to optimal storage tiers. Hot data (frequently accessed) moves to faster tiers (RAM, NVMe), while cold data moves to slower, cheaper tiers (PFS, tape).

Section 4: Context Assimilation (OMNI Format)

Unified scientific data representation. The OMNI file format normalizes HDF5, NetCDF, CSV, JSON into a common schema, enabling seamless access across heterogeneous formats.

Features

Multi-Format Support

  • • HDF5, NetCDF, Zarr
  • • CSV, JSON, Parquet
  • • FITS, ROOT, PDB
  • • 50+ scientific formats

Advanced Capabilities

  • • Semantic metadata: CF conventions, NeXus standards
  • • Lazy loading: Load only requested data
  • • Streaming support: Process data as it arrives

Code Example

#include 

// Create an OMNI file from HDF5
omni::File file;
file.Open("/data/climate.h5", omni::Format::HDF5);

// Query data using unified API
auto temperature = file.GetVariable("temperature");
auto dims = temperature.GetDimensions(); // [time, lat, lon]

// Access data (automatically converted to OMNI format)
auto data = temperature.Read();

// Create new OMNI file
omni::File output;
output.Create("/data/output.omni", omni::Format::OMNI);

// Write data (can be read back as any format)
output.WriteVariable("temperature", data, dims);

// Convert to NetCDF
output.Export("/data/output.nc", omni::Format::NetCDF);

// Query with semantic filters
auto filtered = file.Query({
    {"variable", "temperature"},
    {"time_range", "2024-01-01:2024-12-31"},
    {"lat_range", "-90:90"},
    {"lon_range", "0:360"}
});

OMNI Schema

The OMNI format uses a unified schema based on scientific data conventions:

  • • Variables: Named data arrays with dimensions
  • • Dimensions: Coordinate axes (time, space, etc.)
  • • Attributes: Metadata (units, long_name, etc.)
  • • Groups: Hierarchical organization

Additional Resources

Python API

High-level Python bindings for IOWarp SDK components

Deployment Guide

HPC cluster deployment and configuration