Python API Reference

CLIO Python API: iowarp.clio Module

Module Overview

The iowarp.clio Python module provides high-level access to IOWarp's Context Engineering Engine. It offers a Pythonic interface for context management, data placement, and format assimilation. 

# Python SDK coming soon. Currently available as C++ API via IOWarp Core.

Coming Soon Python SDK coming soon. Currently available as C++ API via IOWarp Core.

ContextInterface Class

The ContextInterface class is the primary interface for interacting with IOWarp's context storage and retrieval system.

Basic Usage

from iowarp.clio import ContextInterface

# Create a context interface instance
ctx = ContextInterface()

# Connect to IOWarp runtime
ctx.connect("localhost:5555")

# Put context (store data with metadata)
import numpy as np
temperature_data = np.random.rand(100, 100).astype(np.float32)
ctx.put("simulation/temperature", temperature_data, metadata={
    "units": "Kelvin",
    "dimensions": ["time", "lat", "lon"],
    "description": "Surface temperature field"
})

# Get context (retrieve data)
result = ctx.get("simulation/temperature",
    time_range="2024-01-01:2024-12-31")

# Access data and metadata
data = result.data
metadata = result.metadata
provenance = result.provenance

Methods

  • connect(host) — Connect to runtime
  • put(key, data, metadata) — Store context
  • get(key, **filters) — Retrieve context
  • delete(key) — Remove context
  • list(prefix) — List available contexts
  • exists(key) — Check if context exists

Query Filters

  • time_range — Temporal filter
  • spatial_range — Spatial bounding box
  • metadata_filter — Custom metadata query
  • version — Specific version number
  • tier — Storage tier preference

AssimilationCtx Class

For format normalization: The AssimilationCtx class handles ingestion and conversion of scientific data formats into IOWarp's unified context representation.

Format Ingestion

from iowarp.clio import AssimilationCtx

# Create assimilation context
assimilation = AssimilationCtx()

# Ingest NetCDF file
context = assimilation.ingest("/data/climate.nc",
    format="netcdf",
    variables=["temperature", "pressure"],
    metadata={
        "experiment": "RCP8.5",
        "model": "CESM2"
    })

# Ingest HDF5 file
context = assimilation.ingest("/data/simulation.h5",
    format="hdf5",
    group="/simulation/output",
    variables=["density", "velocity"])

# Ingest CSV file
context = assimilation.ingest("/data/observations.csv",
    format="csv",
    index_column="timestamp",
    columns=["sensor_1", "sensor_2"])

# Access normalized data
data = context.get_variable("temperature")
dims = context.get_dimensions("temperature")
attrs = context.get_attributes("temperature")

Supported Formats

Scientific:
  • • HDF5
  • • NetCDF
  • • Zarr
  • • FITS
Tabular:
  • • CSV
  • • Parquet
  • • JSON
  • • Excel
Domain-Specific:
  • • ROOT (HEP)
  • • PDB (Bio)
  • • DICOM (Medical)

Data Placement API

Control storage tier placement: Explicitly manage where data is stored across IOWarp's hierarchical storage tiers.

Placement Policies

# Set global placement policy
ctx.set_placement_policy({
    "hot_tier": "nvme",
    "cold_tier": "lustre",
    "prediction": "ml_based"
})

# Set policy for specific context
ctx.set_placement_policy({
    "key": "simulation/temperature",
    "tier": "ram",
    "priority": "high"
})

# Query current placement
placement = ctx.get_placement("simulation/temperature")
print(f"Current tier: {placement.tier}")
print(f"Score: {placement.score}")

# Manually promote/demote data
ctx.promote("simulation/temperature", target_tier="ram")
ctx.demote("simulation/temperature", target_tier="lustre")

Placement Strategies

  • ml_based — ML prediction (default)
  • manual — Explicit user control
  • lru — Least recently used
  • cost_aware — Cost optimization

Storage Tiers

  • ram — Fastest, smallest
  • nvme — High-speed SSD
  • lustre — Parallel file system
  • tape — Archive storage

Batch Operations

Perform multiple operations efficiently with batch APIs that minimize network round-trips and optimize I/O patterns.

Batch Put/Get

# Batch put multiple contexts
contexts = {
    "simulation/temperature": temp_data,
    "simulation/pressure": press_data,
    "simulation/humidity": hum_data
}
results = ctx.batch_put(contexts, metadata={
    "experiment": "2024_run_01"
})

# Batch get multiple contexts
keys = ["simulation/temperature", "simulation/pressure"]
results = ctx.batch_get(keys, time_range="2024-01-01:2024-12-31")

# Process results
for key, result in results.items():
    print(f"{key}: {result.data.shape}")

Performance Benefits

  • • Reduced network overhead through batching
  • • Parallel I/O operations
  • • Optimized data placement decisions

Streaming API

Process data as it arrives: Stream large datasets or real-time data feeds without loading everything into memory.

Streaming Operations

# Stream data into context
def data_generator():
    for i in range(1000):
        yield np.random.rand(100, 100)

ctx.stream_put("simulation/stream", data_generator(),
    chunk_size=100,
    metadata={"source": "realtime_sensor"})

# Stream data from context
for chunk in ctx.stream_get("simulation/stream",
                            chunk_size=100):
    # Process chunk
    process(chunk)

# Stream with callbacks
def on_chunk(chunk, metadata):
    print(f"Received chunk: {chunk.shape}")

ctx.stream_get("simulation/stream",
               callback=on_chunk,
               chunk_size=100)

Error Handling

The Python API provides comprehensive error handling with custom exceptions for different failure modes.

Exception Types

from iowarp.clio import (
    ContextInterface,
    ContextNotFoundError,
    ConnectionError,
    PlacementError,
    FormatError
)

try:
    ctx = ContextInterface()
    ctx.connect("localhost:5555")
    result = ctx.get("nonexistent/key")
except ConnectionError as e:
    print(f"Failed to connect: {'{'}e{'}'}")
except ContextNotFoundError as e:
    print(f"Context not found: {'{'}e{'}'}")
except PlacementError as e:
    print(f"Placement failed: {'{'}e{'}'}")
except FormatError as e:
    print(f"Format error: {'{'}e{'}'}")

Exception Hierarchy

  • IOWarpError — Base exception
  • ConnectionError — Network/runtime errors
  • ContextNotFoundError — Missing context
  • PlacementError — Storage tier errors
  • FormatError — Data format errors

Error Recovery

  • • Automatic retry with exponential backoff
  • • Connection pooling and failover
  • • Graceful degradation

Additional Resources

SDK Reference

C++ SDK documentation for advanced use cases

FAQ

Common questions and troubleshooting