Deployment Guide

Deploy IOWarp on HPC Clusters

Spack Deployment

Spack is the recommended package manager for HPC environments. It provides optimized builds, dependency management, and multi-node deployment capabilities.

Single-Node Installation

# Install IOWarp
spack install iowarp

# Load IOWarp environment
spack load iowarp

# Verify installation
clio status

Multi-Node Deployment

# Install on all nodes
parallel-ssh -i -h hostfile -x "-o SendEnv=PATH" \
  "spack install iowarp && spack load iowarp"

# Start runtime on all nodes
parallel-ssh -i -h hostfile -x "-o SendEnv=PATH" \
  "clio deploy"

# Verify all nodes are running
parallel-ssh -i -h hostfile \
  "clio status"

Spack Variants

Customize your IOWarp installation with Spack variants: 

# With CUDA support
spack install iowarp +cuda cuda_arch=70

# With MPI support
spack install iowarp +mpi

# With specific compiler
spack install iowarp %gcc@12

# With debug symbols
spack install iowarp +debug

# Combined variants
spack install iowarp +cuda +mpi %gcc@12

YAML Configuration Reference

Complete reference for wrp_conf.yaml with all available parameters and options.

Full Configuration Example

compose:
  - mod_name: wrp_cte_core
    pool_name: wrp_cte
    pool_query: local
    pool_id: 512.0
    storage:
      # Critical tier: RAM (fastest)
      - path: "ram::cte_ram_tier1"
        bdev_type: "ram"
        capacity_limit: "16GB"
        score: 0.0
      
      # Working tier: NVMe SSD
      - path: "nvme::cte_nvme_tier1"
        bdev_type: "nvme"
        capacity_limit: "1TB"
        score: 5.0
        device: "/dev/nvme0n1"
      
      # Reference tier: Parallel File System
      - path: "lustre::cte_lustre_tier1"
        bdev_type: "file"
        capacity_limit: "100TB"
        score: 50.0
        mount_point: "/lustre/iowarp"
      
      # Archive tier: Async I/O
      - path: "tape::cte_tape_tier1"
        bdev_type: "async"
        capacity_limit: "1PB"
        score: 100.0
        backend: "ltfs"

# Network configuration
network:
  zmq_port: 5555
  zmq_bind_address: "0.0.0.0"
  enable_tls: false
  tls_cert: "/etc/iowarp/cert.pem"
  tls_key: "/etc/iowarp/key.pem"

# Performance tuning
performance:
  num_workers: 8
  queue_depth: 64
  buffer_size: "1MB"
  prefetch_enabled: true
  prefetch_size: "100MB"

# Logging
logging:
  level: "info"
  file: "/var/log/iowarp/runtime.log"
  rotation_size: "100MB"
  max_files: 10

compose Section

  • mod_name: Runtime module name
  • pool_name: Memory pool identifier
  • pool_query: local or remote
  • pool_id: Unique pool ID (float)

storage Section

  • path: Tier identifier
  • bdev_type: ram, nvme, file, async
  • capacity_limit: Max capacity
  • score: Placement priority

Storage Tiers

RAM

Fastest tier, smallest capacity

bdev_type: "ram"

NVMe

High-speed SSD storage

bdev_type: "nvme"
device: "/dev/nvme0n1"

File-Based

Parallel file systems (Lustre, GPFS)

bdev_type: "file"
mount_point: "/lustre/iowarp"

Async I/O

Tape, cloud, object storage

bdev_type: "async"
backend: "ltfs"

Performance Tuning

Optimize IOWarp performance for your specific workload and hardware configuration.

Worker Threads

Control the number of worker threads using the CHIMAERA_NUM_WORKERS environment variable: 

# Set number of workers (default: number of CPU cores)
export CHIMAERA_NUM_WORKERS=16

# Or in wrp_conf.yaml
performance:
  num_workers: 16

Recommendation: Set to number of CPU cores for CPU-bound workloads, or number of I/O devices for I/O-bound workloads.

DPE (Data Placement Engine) Strategies

ML-Based (Default)

Uses machine learning models to predict access patterns:

placement_strategy: "ml_based"

LRU (Least Recently Used)

Simple eviction policy based on access recency:

placement_strategy: "lru"

Buffer Sizes and Queue Depths

performance:
  # I/O buffer size per operation
  buffer_size: "1MB"  # Options: 64KB, 256KB, 1MB, 4MB
  
  # Queue depth for async operations
  queue_depth: 64  # Increase for high I/O parallelism
  
  # Prefetch configuration
  prefetch_enabled: true
  prefetch_size: "100MB"  # Amount to prefetch ahead

Tuning Tips: Increase buffer_size for large sequential I/O, increase queue_depth for random I/O patterns, enable prefetch for sequential access patterns.

Container Deployment

Deploy IOWarp using Docker or Kubernetes for containerized environments.

Docker Deployment

# Run IOWarp container
docker run -d \
  --name iowarp-runtime \
  --shm-size=8g \
  --ipc=shareable \
  -p 5555:5555 \
  -v ./wrp_conf.yaml:/etc/iowarp/wrp_conf.yaml:ro \
  iowarp/iowarp:latest

Kubernetes Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: iowarp-runtime
spec:
  replicas: 1
  selector:
    matchLabels:
      app: iowarp
  template:
    metadata:
      labels:
        app: iowarp
    spec:
      containers:
      - name: iowarp
        image: iowarp/iowarp:latest
        ports:
        - containerPort: 5555
        volumeMounts:
        - name: config
          mountPath: /etc/iowarp/wrp_conf.yaml
          subPath: wrp_conf.yaml
        resources:
          requests:
            memory: "8Gi"
          limits:
            memory: "16Gi"
      volumes:
      - name: config
        configMap:
          name: iowarp-config

Multi-Node Configuration

Configure IOWarp for multi-node HPC deployments with shared storage and network coordination.

Hostfile Setup

# hostfile format
node001.cluster.local
node002.cluster.local
node003.cluster.local
node004.cluster.local

Network Configuration

# wrp_conf.yaml for multi-node
network:
  zmq_port: 5555
  zmq_bind_address: "0.0.0.0"
  
  # High-speed interconnect (InfiniBand, Omni-Path)
  interconnect: "ib0"
  interconnect_port: 5556
  
  # Discovery service
  discovery:
    enabled: true
    method: "zeroconf"  # or "static"
    static_nodes:
      - "node001.cluster.local:5555"
      - "node002.cluster.local:5555"

Multi-Node Best Practices

  • • Use high-speed interconnects (InfiniBand) for data transfer
  • • Configure shared storage (Lustre, GPFS) for persistent data
  • • Enable discovery service for automatic node detection
  • • Use job schedulers (Slurm, PBS) for coordinated startup

Additional Resources

Getting Started

Quick start guide and basic configuration

FAQ

Troubleshooting deployment issues