Contents: What is Apache Iggy? · Features · Architecture · Clustering · Version · Supported languages SDK · CLI · Web UI · Connectors · Kafka gateway · Model Context Protocol · Docker · Kubernetes · Configuration · Run the server · Try the CLI · Examples · SDK · Benchmarks · Contributing

What is Apache Iggy?

Iggy is a persistent message streaming platform written in Rust, supporting QUIC, WebSocket, TCP (custom binary specification) and HTTP (regular REST API) transport protocols, capable of processing millions of messages per second at ultra-low latency.

Iggy provides exceptionally high throughput and performance while utilizing minimal computing resources.

This is not yet another extension running on top of existing infrastructure, such as Kafka or SQL database.

Iggy is a persistent message streaming log built from the ground up using low-level I/O with thread-per-core shared nothing architecture, io_uring and compio for maximum speed and efficiency.

The name is an abbreviation for the Italian Greyhound - small yet extremely fast dogs, the best in their class. See the lovely Fabio & Cookie ❤️

Features

• Highly performant, persistent append-only log for message streaming
• Very high throughput for both writes and reads
• Low latency and predictable resource usage thanks to the Rust compiled language (no GC) and io_uring.
• User authentication and authorization with granular permissions and Personal Access Tokens (PAT)
• Support for multiple streams, topics and partitions
• Support for multiple transport protocols (QUIC, WebSocket, TCP, HTTP)
• Fully operational RESTful API which can be optionally enabled
• Available client SDK in multiple languages
• Thread per core shared nothing design together with io_uring guarantee the best possible performance on modern Linux systems.
• Works directly with binary data, avoiding enforced schema and serialization/deserialization overhead
• Custom zero-copy (de)serialization, which greatly improves the performance and reduces memory usage.
• Configurable server features (e.g. caching and transport protocols), plus per-topic segment size, durability and flush thresholds
• Server-side storage of consumer offsets
• Multiple ways of polling the messages:
• By offset (using the indexes)
• By timestamp (using the time indexes)
• First/Last N messages
• Next N messages for the specific consumer
• Optional poll auto-commit; processing guarantees depend on application processing and offset-commit ordering
• Consumer groups providing the message ordering and horizontal scaling across the connected clients
• Message expiry with auto deletion based on the configurable retention policy
• Additional features such as server side message deduplication
• Multi-tenant support via abstraction of streams which group topics
• TLS support for all transport protocols (TCP, WebSocket, QUIC, HTTPS)
• Connectors - sinks, sources and data transformations based on the custom Rust plugins
• Model Context Protocol - provide context to LLM with MCP server
• Optional server-side as well as client-side data encryption using AES-256-GCM, server-side with a separate key per stream
• Optional metadata support in the form of message headers
• Server-side message compression is not supported yet. Topic compression values are reserved for future disk/network compression support; use message headers for manual compression today (see examples/rust/src/message-headers/message-compression).
• Optional data backups and archiving to disk or S3 compatible cloud storage (e.g. AWS S3)
• Prometheus metrics for the server and connectors runtime, plus OpenTelemetry logs & traces in the connectors runtime. Server OTLP export is unavailable pending runtime integration.
• Built-in CLI to manage the streaming server installable via cargo install iggy-cli
• Built-in benchmarking app to test the performance
• Single binary deployment without an external broker or database; dynamically linked builds still require operating-system libraries
• Running as a single node or as a cluster, with data replication based on Viewstamped Replication (VSR)

Architecture

This is the high-level architecture of the Iggy message streaming server, where extremely high performance and ultra low and stable tail latencies are the primary goals. The server is designed to handle high throughput and very low latency (sub-millisecond tail latencies), making it suitable for real-time applications. For more details, please refer to the documentation.

Clustering

Clustering is built into the standard iggy-server binary, with no separate build or feature flag. The same server runs as a single node or as a cluster, and cluster.enabled in the configuration chooses which. Clustering is disabled by default.

Replicas stay consistent using Viewstamped Replication (VSR), which commits on a quorum and elects a new primary when the current one fails. See the clustering documentation for configuration and deployment.

Version

The official releases follow semver (for example server-0.9.0), and the Docker images also carry the latest tag (apache/iggy:latest).

The server, SDKs and CLI are versioned in