Streams Fundamentals

Before Redis 5.0, developers had to choose between the speed of Pub/Sub (which had no persistence) and the reliability of Lists (which were not designed for complex consumption). Enter Redis Streams.

A Stream is an append-only log data structure, similar to Apache Kafka, but implemented with the in-memory speed of Redis.

1. The Append-Only Log

At its core, a Stream is just an infinite list of messages.

• Ordered: Messages are stored in the order they are received.
• Immutable: Once a message is written, it (generally) doesn’t change.
• Persistent: Unlike Pub/Sub, messages are stored on disk (via RDB/AOF) and survive server restarts.

1.1 Anatomy of a Message

Every message in a stream consists of:

1. ID: A unique identifier, typically a timestamp (e.g., 1623456789123-0).
2. Fields: One or more key-value pairs (e.g., sensor_id="A1", temp="22.5").

2. Basic Commands

Adding Data (XADD)

The XADD command appends a new entry to the stream. You almost always use * to let Redis auto-generate the ID.

XADD mystream * sensor_id "A1" temp "22.5"
# Returns: "1623456789123-0"

Reading History (XRANGE)

Since a stream is persistent, you can query it by time range.

# Read from beginning (-) to end (+)
XRANGE mystream - +

# Read just the first 2 entries
XRANGE mystream - + COUNT 2

---

3. Interactive: Stream Internals

Visualize how XADD appends data to the log and how Redis efficiently stores it internally.

Stream Length: 0 Memory: 0B

Append-Only Log

Stream is empty

Internal Structure (Radix Tree + Listpacks)

Root

---

4. Under the Hood: Radix Trees & Listpacks

Redis Streams are not implemented as simple linked lists. To save memory, they use a hybrid data structure called a Radix Tree (or Rax).

4.1 The Rax (Radix Tree)

A Radix Tree is a compressed prefix tree. Instead of storing every ID explicitly, it shares common prefixes.

• Example: IDs 1623456789000-0 and 1623456789000-1 share the prefix 1623456789000.
• Efficiency: This drastically reduces the memory overhead for storing timestamps.

4.2 Listpacks

Inside each node of the Radix Tree, Redis doesn’t store a single message. Instead, it stores a Listpack (a tightly packed list of binary data) containing multiple messages (typically 50-100).

• Pointer Chasing: This reduces the number of pointers Redis needs to follow.
• CPU Cache: Packed data is friendlier to CPU caches, improving iteration speed.

[!NOTE] This hybrid approach allows Redis Streams to store millions of messages with significantly less memory than a standard Linked List or Hash.

---

5. Code Examples: Producer

package main

import (
    "context"
    "fmt"
    "github.com/redis/go-redis/v9"
)

func main() {
    ctx := context.Background()
    rdb := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })

    // XADD mystream * sensor_id A1 temp 22.5
    id, err := rdb.XAdd(ctx, &redis.XAddArgs{
        Stream: "mystream",
        Values: map[string]interface{}{
            "sensor_id": "A1",
            "temp":      22.5,
        },
    }).Result()

    if err != nil {
        panic(err)
    }

    fmt.Println("Produced message:", id)
}

6. Summary

Redis Streams provide a high-performance, persistent log. By using XADD and XRANGE, you can treat Redis as a lightweight time-series database or an event store. However, reading raw ranges is hard to scale. In the next chapter, we’ll see how Consumer Groups solve this.