Platform Overview
CLIO Core's four integrated engines bridge AI agents with scientific infrastructure.
Why IOWarp?
Traditional AI systems struggle with context continuity — agents lose track of prior interactions, fail to coordinate, and lack observability. In scientific computing: petabyte-scale datasets, heterogeneous formats (HDF5, NetCDF, ROOT), real-time streams, multi-site collaborations.
IOWarp solves through infrastructure-first context engineering:
- ▸ Persistent Context: survives beyond single interactions, spanning months-long campaigns
- ▸ Multi-Agent Coordination: specialized agents share context efficiently
- ▸ Production Observability: real-time insights into agent reasoning
- ▸ Standards-Based: MCP, OpenTelemetry, CF conventions, NeXus
- ▸ Scientific-Native: Slurm, PBS, Lustre, GPFS integration
The Scientific Computing Gap
80%+ researcher time wasted on data prep. Need:
Multi-Modal Data Integration
Unified access across heterogeneous formats and sources
Hierarchical Storage
GPU HBM → NVMe → Parallel FS → Cloud/Tape
Quality Assurance
Conservation laws, provenance tracking, validation
Scale
Petabyte datasets, multi-site collaborations
Platform Architecture
Three layers — from agent to infrastructure — with CLIO Core orchestrating context across the entire pipeline.
CLIO Core — The Runtime Platform
The umbrella runtime housing all four engines: Ingest, Transfer, Explore, and Plugin. CLIO Core provides the unified context orchestration layer that agents build on.
CLIO Ingest — Data Ingestion Engine
"The Way In"
Multi-Modal Data Ingestion
Supports:
- • HDF5, NetCDF, Zarr
- • Sensor streams
- • Literature
- • Telemetry
Format Normalization
50+ scientific formats, plugin-based architecture, xarray-like representation
Semantic Enrichment
Automatic extraction of units, coordinates, provenance, quality metadata
Dynamic Windowing
Temporal/spatial/variable selection for efficient data access
CLIO Transfer — Intelligent Data Movement
"The Mover"
Hierarchical Tiers
- Critical: GPU HBM, TB/s
- Working: NVMe, GB/s
- Reference: PFS, PBs
- Archive: Tape, EBs
Agent-Aware Features
- • Predictive staging
- • Memory pressure management
- • Multi-agent dedup
- • RDMA/GPU Direct/CXL
MegaMmap SC'24 Performance
CLIO Explore — Context Search & Exploration
"The Finder"
CLIO Query
Natural language interface for petabyte-scale data exploration
Query Translation
Converts natural language to data source ID, method selection, output spec
Multi-Agent Orchestration
Coordinates telemetry + analysis + visualization + synthesis agents
CLIO Plugin — Platform Integration Layer
"The Connector"
HPC
Slurm, PBS, LSF
Instruments
Detectors, telescopes, sensors
Storage
Lustre, GPFS, S3, Ceph, tape
Security
Execution sandboxing, access control
Cross-Layer Data Flow
Climate Model Analysis Walkthrough
Query Translation (CLIO Explore)
Natural language query converted to data source identification and method selection
Context Assembly (CLIO Ingest)
20TB dataset → 200GB relevant subset through semantic filtering
Hierarchical Staging (CLIO Transfer)
Predictive data movement across storage tiers
Agent Analysis
Multi-agent coordination for analysis execution
Result Synthesis
Aggregation and presentation of results
Performance: 45 seconds end-to-end vs hours manual (projected)
Context as First-Class Infrastructure
Context bundles persist across sessions, enabling long-running scientific campaigns:
bundle_id: climate_analysis_2024_q1
version: 1.2.3
created: 2024-01-15T10:30:00Z
context_layers:
- layer: metadata
source: /data/climate/models/cesm2
format: netcdf
variables: [temperature, precipitation, pressure]
temporal_range: [2020-01-01, 2024-01-01]
- layer: provenance
workflow_id: wf_789
agents: [telemetry_agent, analysis_agent]
transformations: [normalize, validate, aggregate]
- layer: quality
checks: [conservation_laws, bounds_check]
status: passed
- layer: semantic
units: {temperature: "K", precipitation: "mm/day"}
coordinates: {lat: [-90, 90], lon: [-180, 180]}
attributes: {title: "CESM2 Climate Model Output"}Performance by Design
Context Retrieval
p99 working tier
<100ms reference
Agent Coordination
RDMA messaging
Data Movement
faster (MegaMmap SC'24)
Scalability
Petabyte datasets
Hundreds of concurrent agents