Redis Modules: RedisSearch, RedisJSON and RedisTimeSeries

Chapter 29: Redis Modules — RedisSearch, RedisJSON, and RedisTimeSeries

29.1 The Module System

Redis Modules (introduced in Redis 4.0) allow developers to extend Redis with shared libraries (.so files). A module can register new commands and new data types, hook into the keyspace, subscribe to events, and run background timers—all while executing inside the Redis process with the same performance characteristics as built-in commands.

29.1.1 Loading and Managing Modules

# redis.conf — static loading at startup
loadmodule /usr/lib/redis/modules/redisearch.so
loadmodule /usr/lib/redis/modules/rejson.so
loadmodule /usr/lib/redis/modules/redistimeseries.so
loadmodule /usr/lib/redis/modules/redisbloom.so

# Runtime loading (no restart required)
MODULE LOAD /path/to/module.so [arg1 arg2 ...]

# Inspect loaded modules
MODULE LIST
# 1) 1) "name"   2) "search"
#    3) "ver"    4) (integer) 20601
#    5) "path"   6) "/usr/lib/redis/modules/redisearch.so"
#    7) "args"   8) (empty array)

# Unload (fails if keys of that module's type exist)
MODULE UNLOAD search

29.1.2 What the Module API Provides

• Command registration: RedisModule_CreateCommand — new commands run at the same priority as built-ins
• Custom data types: custom RDB serialization and AOF rewrite hooks for full persistence compatibility
• Keyspace access: read/write keys, check/set TTLs, subscribe to keyspace events
• Blocking clients: RedisModule_BlockClient enables BLPOP-style blocking commands
• Background threads: RedisModule_CreateTimer for async processing without blocking the main thread
• Command filters: pre/post interception hooks for all commands

29.2 RedisSearch — Full-Text and Vector Search

RedisSearch is the core module of Redis Stack. It builds inverted indexes and vector indexes on top of Redis Hash or JSON data, enabling sub-millisecond full-text search and approximate nearest-neighbor (ANN) vector queries.

29.2.1 Creating an Index

# Index on Hash type
FT.CREATE product_idx
  ON HASH
  PREFIX 1 product:
  SCHEMA
    name        TEXT    WEIGHT 5.0 NOSTEM
    description TEXT    WEIGHT 1.0
    price       NUMERIC SORTABLE
    category    TAG     SEPARATOR ","
    stock       NUMERIC
    embedding   VECTOR  HNSW 6
                TYPE FLOAT32
                DIM 128
                DISTANCE_METRIC COSINE

# Index on JSON type
FT.CREATE user_idx
  ON JSON
  PREFIX 1 user:
  SCHEMA
    $.name      AS name  TEXT
    $.age       AS age   NUMERIC SORTABLE
    $.tags[*]   AS tags  TAG
    $.bio       AS bio   TEXT

# Inspect an index
FT.INFO product_idx

29.2.2 Search Query Syntax

# Full-text search
FT.SEARCH product_idx "redis database"

# Field filters
FT.SEARCH product_idx "@category:{database} @price:[10 100]"

# Boolean logic
FT.SEARCH product_idx "(redis | memcached) @price:[0 50]"

# Sort and paginate
FT.SEARCH product_idx "cache"
  SORTBY price ASC
  LIMIT 0 20

# Return specific fields only
FT.SEARCH product_idx "redis"
  RETURN 3 name price category

# Highlight matching terms
FT.SEARCH product_idx "fast cache"
  HIGHLIGHT FIELDS 1 description
  TAGS "<b>" "</b>"

# Fuzzy search (edit distance ≤ 1)
FT.SEARCH product_idx "%reddis%"

# Prefix search
FT.SEARCH product_idx "redi*"

29.2.3 How the Inverted Index Works

Source documents:
  product:1 → name = "Redis Database Guide"
  product:2 → name = "Redis Performance Tips"
  product:3 → name = "PostgreSQL Guide"

Inverted index (after stemming and tokenization):
  "redis"    → [(product:1, TF=0.33), (product:2, TF=0.33)]
  "databas"  → [(product:1, TF=0.33), (product:3, TF=0.33)]  ← stemmed
  "guid"     → [(product:1, TF=0.33), (product:3, TF=0.33)]  ← stemmed
  "perform"  → [(product:2, TF=0.33)]

Query "redis guide":
  "redis" posting list ∩ "guid" posting list → product:1
  BM25 scoring: product:1 (2.5), product:3 (0.8, partial match)

29.2.4 Aggregation Queries

# Count products per category, compute average price
FT.AGGREGATE product_idx "*"
  GROUPBY 1 @category
  REDUCE COUNT 0 AS count
  REDUCE AVG 1 @price AS avg_price
  SORTBY 2 @count DESC

# Sample output:
# 1) "category" "database"  "count" "45"  "avg_price" "35.6"
# 2) "category" "cache"     "count" "12"  "avg_price" "20.1"

# Time-based aggregation (items created in the last 7 days)
FT.AGGREGATE product_idx "@created_at:[7_days_ago +inf]"
  APPLY "@created_at - (@created_at % 86400)" AS day
  GROUPBY 1 @day
  REDUCE COUNT 0 AS daily_count
  SORTBY 2 @day ASC

29.2.5 Vector Search (KNN)

Vector search enables semantic similarity queries—finding the most similar documents to a query embedding.

Index type comparison:

# HNSW vector index — recommended for large datasets
FT.CREATE product_idx ON HASH PREFIX 1 product:
  SCHEMA
    name      TEXT
    embedding VECTOR HNSW 12
              TYPE FLOAT32
              DIM 128
              DISTANCE_METRIC COSINE
              M 16                # Max neighbors per layer (accuracy vs. memory)
              EF_CONSTRUCTION 200 # Build-time search width (index quality)
              EF_RUNTIME 10       # Query-time search width (speed vs. accuracy)

# KNN search: find 10 most similar products
FT.SEARCH product_idx
  "*=>[KNN 10 @embedding $vec AS score]"
  PARAMS 2 vec "\x00\x01\x02..."
  RETURN 3 name score price
  SORTBY score
  DIALECT 2

# Python: semantic product search using OpenAI embeddings
import redis, numpy as np
from redis.commands.search.query import Query

r = redis.Redis()

def index_product(product_id: str, name: str, embedding: np.ndarray):
    """Store a product with its embedding vector."""
    r.hset(f"product:{product_id}", mapping={
        "name": name,
        "embedding": embedding.astype(np.float32).tobytes()
    })

def search_similar_products(query_embedding: np.ndarray, top_k: int = 10):
    """Find the top-k most similar products by vector similarity."""
    vec_bytes = query_embedding.astype(np.float32).tobytes()
    results = r.ft("product_idx").search(
        Query("*=>[KNN $k @embedding $vec AS score]")
            .sort_by("score")
            .paging(0, top_k)
            .return_fields("name", "score", "price")
            .dialect(2),
        query_params={"k": top_k, "vec": vec_bytes}
    )
    return [(doc.name, float(doc.score)) for doc in results.docs]

29.2.6 RedisSearch vs. Elasticsearch

29.3 RedisJSON — Native JSON Storage

29.3.1 Why Not Just Store JSON as a String

The naive approach: SET user:1000 '{"name":"Alice","age":30}'

Problems:

• Updating a single field requires: GET → deserialize → modify → serialize → SET (2 round trips minimum)
• No way to index individual JSON fields without a custom solution
• Concurrent updates risk overwriting each other (requires WATCH + transaction)
• Value must be fully read/written even for tiny changes

RedisJSON solves these by providing field-level access, JSONPath queries, and native indexing integration with RedisSearch.

29.3.2 Core Operations

# Store a JSON document
JSON.SET user:1000 $ '{"name":"Alice","age":30,"tags":["admin","user"],"address":{"city":"Beijing"}}'

# Read the full document
JSON.GET user:1000

# Read with JSONPath ($ = root)
JSON.GET user:1000 $.name           # Returns: ["Alice"]
JSON.GET user:1000 $.address.city   # Returns: ["Beijing"]
JSON.GET user:1000 $.tags[0]        # Returns: ["admin"]

# Read multiple paths in one call
JSON.GET user:1000 $.name $.age
# Returns: {"$.name":["Alice"],"$.age":[30]}

# Update a field (only this field is written, no full document read-write)
JSON.SET user:1000 $.age 31
JSON.SET user:1000 $.address.city "Shanghai"

# Numeric operations
JSON.NUMINCRBY user:1000 $.age 1    # Returns: [32]

# Array operations
JSON.ARRAPPEND user:1000 $.tags '"superuser"'  # Returns: [3] (new length)
JSON.ARRLEN user:1000 $.tags                   # Returns: [3]
JSON.ARRPOP user:1000 $.tags 0                 # Returns: ["admin"] (pop index 0)
JSON.ARRINSERT user:1000 $.tags 0 '"root"'     # Insert at index 0

# Delete a field
JSON.DEL user:1000 $.address.city

# Type inspection
JSON.TYPE user:1000 $             # ["object"]
JSON.TYPE user:1000 $.tags        # ["array"]
JSON.TYPE user:1000 $.age         # ["integer"]

# Document size
JSON.STRLEN user:1000 $.name      # [5] — string length in bytes
JSON.OBJLEN user:1000 $           # [4] — number of top-level keys
JSON.ARRLEN user:1000 $.tags      # [2] — array element count

29.3.3 JSONPath Syntax Reference

$              # Root node
$.field        # Property access
$[0]           # Array index 0 (zero-based)
$[-1]          # Last array element
$[0:3]         # Array slice: elements 0, 1, 2
$.*            # All direct children
$..field       # Recursive descent — find all "field" properties at any depth
$[?(@.age > 18)]              # Filter: elements where age > 18
$[?(@.role == "admin")]       # Filter: elements where role equals "admin"
$[?(@.active == true)]        # Filter: boolean comparison

# Examples
JSON.GET users:list $[?(@.age >= 18)].name   # Names of all adult users
JSON.GET config $..timeout                    # All timeout values, anywhere in doc
JSON.GET order:1 $.items[*].price             # All item prices in an order

29.3.4 Integration with RedisSearch

# Create search index on JSON documents
FT.CREATE user_idx ON JSON PREFIX 1 user:
  SCHEMA
    $.name    AS name  TEXT
    $.age     AS age   NUMERIC SORTABLE
    $.tags[*] AS tags  TAG

# Add some documents
JSON.SET user:1 $ '{"name":"Alice","age":28,"tags":["admin"]}'
JSON.SET user:2 $ '{"name":"Bob","age":35,"tags":["user","reader"]}'

# Search across JSON fields
FT.SEARCH user_idx "@age:[25 35] @tags:{admin}"
  RETURN 2 $.name $.age

29.3.5 Internal Implementation

RedisJSON registers a custom Redis data type backed by a RapidJSON document tree in memory. This is not a string—it's a fully parsed JSON document maintained as a C++ object tree.

Benefits of this approach:

• Field-level access is O(path depth), not O(document size)
• JSONPath evaluation runs directly on the in-memory tree
• RedisSearch can index individual fields without parsing the full document

RDB persistence: RedisJSON implements the RedisModule_SaveString family of callbacks to serialize the JSON tree into a compact binary format in RDB snapshots. The module is fully compatible with RDB and AOF persistence.

29.4 RedisTimeSeries — Time Series Storage

29.4.1 Why Dedicated Time Series Storage

Time series data has a distinct access pattern:

• Writes: append-only, almost never update historical data
• Reads: time range queries with aggregation (avg, max, min over intervals)
• Volume: IoT and monitoring workloads generate millions of data points per second

The naive Sorted Set approach (ZADD sensor:temp <timestamp> <value>) has serious drawbacks:

• Each data point costs ~60–80 bytes in a Sorted Set (ziplist threshold exceeded quickly)
• No built-in time-based aggregation
• No compression

RedisTimeSeries compresses data by 4–10x and provides native aggregation.

29.4.2 Core Operations

# Create a time series key
TS.CREATE temperature
  RETENTION 86400000          # Keep data for 24 hours (ms), 0 = forever
  CHUNK_SIZE 4096             # Memory chunk size in bytes (default: 4096)
  ENCODING COMPRESSED         # Use compression (default), or UNCOMPRESSED
  DUPLICATE_POLICY LAST       # How to handle duplicate timestamps
  LABELS sensor_id 1 location "floor-3" unit "celsius"

# Add data points (* = current Unix time in milliseconds)
TS.ADD temperature * 23.5
TS.ADD temperature 1716000000000 24.1  # Explicit timestamp

# Batch insert across multiple keys
TS.MADD
  temperature 1716000001000 23.8
  humidity    1716000001000 65.2
  pressure    1716000001000 1013.2

# Query a time range
TS.RANGE temperature - +              # All data (- = earliest, + = latest)
TS.RANGE temperature 1716000000000 1716003600000  # Last hour

# Time range with aggregation (1-minute averages)
TS.RANGE temperature - +
  AGGREGATION avg 60000               # 60000ms = 1 minute

# Available aggregation functions:
# avg, sum, min, max, range, count, first, last, std.p, std.s, var.p, var.s

# Get the latest value
TS.GET temperature

# Reverse range query (newest first)
TS.REVRANGE temperature - + LIMIT 0 100

29.4.3 Automatic Downsampling (Compaction Rules)

# First, create the target key for the downsampled data
TS.CREATE temperature:hourly RETENTION 2592000000  # Keep 30 days

# Create a compaction rule: raw 1-minute data → 1-hour averages
TS.CREATERULE
  temperature           # Source key
  temperature:hourly    # Destination key
  AGGREGATION avg 3600000  # Aggregate every 3600000ms = 1 hour

# Query hourly averages
TS.RANGE temperature:hourly - + LIMIT 0 24

# Multi-rule setup: raw → 1min avg → 1hour avg → 1day avg
TS.CREATE temperature:1min RETENTION 604800000   # 7 days
TS.CREATE temperature:1hour RETENTION 2592000000 # 30 days
TS.CREATE temperature:1day  RETENTION 0          # Forever

TS.CREATERULE temperature temperature:1min  AGGREGATION avg 60000
TS.CREATERULE temperature temperature:1hour AGGREGATION avg 3600000
TS.CREATERULE temperature temperature:1day  AGGREGATION avg 86400000

29.4.4 Multi-Key Queries with MRANGE

# Query all time series with matching labels
TS.MRANGE - + FILTER sensor_id=1
TS.MRANGE - + AGGREGATION avg 60000 FILTER location="floor-3"

# Group by label and aggregate
TS.MRANGE - + AGGREGATION avg 3600000
  GROUPBY location REDUCE avg
  FILTER unit=celsius

# Update labels on an existing key
TS.ALTER temperature LABELS sensor_id 1 location "floor-3" unit "celsius" status "active"

29.4.5 Compression Algorithm Deep Dive

RedisTimeSeries applies two compression layers to minimize memory usage:

Timestamp compression (Delta-of-Delta):

Raw timestamps (ms):   1716000000, 1716000060, 1716000120, 1716000180
Deltas:                60, 60, 60
Delta-of-Delta:        0, 0, 0    ← near-zero for regular intervals

For perfectly regular intervals (e.g., one point per second), delta-of-delta values are all zero and compress to near-nothing with Fibonacci encoding.

Value compression (Gorilla XOR):

Raw float values:  23.5, 23.6, 23.4, 23.7
XOR of adjacent:   Adjacent values share sign and exponent bits
Result:            Only differing mantissa bits need to be stored

Values with small changes between samples compress to 2–4 bits per point.

Measured compression ratio:

• Regular-interval time series: ~1.5 bytes/point (vs. 16 bytes uncompressed)
• Irregular, random float values: ~8 bytes/point
• Typical monitoring workload: 4–6x compression

29.5 RedisBloom — Probabilistic Data Structures

29.5.1 Bloom Filter (BF)

A Bloom filter answers: "Has this element been added before?" It has no false negatives (if it says no, the element was definitely never added) but can have false positives (it might say yes when the element was never added).

# Create with explicit false-positive rate and capacity
BF.RESERVE users_seen 0.001 10000000
# Error rate: 0.1%, initial capacity: 10 million elements

# Add elements (auto-creates with 1% error rate, capacity 100 if key doesn't exist)
BF.ADD users_seen "user:12345"    # Returns: 1 (newly added)
BF.ADD users_seen "user:12345"    # Returns: 0 (already present)

# Batch add
BF.MADD users_seen "user:1" "user:2" "user:3"

# Query
BF.EXISTS users_seen "user:12345"  # Returns: 1 (probably exists)
BF.EXISTS users_seen "user:99999"  # Returns: 0 (definitely not present)

# Batch query
BF.MEXISTS users_seen "user:1" "user:2" "user:unknown"
# Returns: 1, 1, 0

# Inspect filter stats
BF.INFO users_seen
# Capacity: 10000000
# Size: 13807669   ← bytes used
# Number of filters: 1
# Number of items inserted: 4
# Expansion rate: 2

29.5.2 Scalable Bloom Filters

Standard Bloom filters have a fixed capacity; as they fill up, the false positive rate rises sharply. RedisBloom's BF.ADD auto-scales by adding new filter layers when the current layer reaches capacity:

# EXPANSION parameter controls growth factor (default: 2 = doubles each time)
BF.RESERVE scalable_bf 0.01 1000 EXPANSION 2
# Starts at 1000 capacity, grows to 2000, 4000, 8000, ...
# False positive rate stays stable across layers

Trade-off: each expansion adds a layer, and queries must check all layers. Performance degrades slightly with many expansions.

29.5.3 Cuckoo Filter (CF) — Supports Deletion

Bloom filters cannot support element deletion (removing an element might accidentally clear bits shared with other elements). Cuckoo filters solve this:

CF.RESERVE items_cf 1000000      # Reserve for ~1M elements

CF.ADD    items_cf "item:123"    → 1
CF.EXISTS items_cf "item:123"    → 1

# Deletion — impossible with a Bloom filter!
CF.DELETE items_cf "item:123"    → 1
CF.EXISTS items_cf "item:123"    → 0  (correctly reports absent)

Cuckoo filters use two hash tables and a "kicking out" eviction strategy that keeps memory usage comparable to Bloom filters while enabling deletion. Caveat: insertions can fail when the filter is at high load (> 95% capacity).

29.5.4 Count-Min Sketch — Frequency Estimation

# Track approximate frequency of items (top-N hot terms, IP hit counts)
CMS.INITBYDIM freq_sketch 2000 5  # Width 2000, depth 5
CMS.INCRBY freq_sketch "redis" 1
CMS.INCRBY freq_sketch "redis" 1
CMS.QUERY freq_sketch "redis"     # Returns: 2 (approximate, may over-count)
CMS.QUERY freq_sketch "python"    # Returns: 0

# Merge multiple sketches
CMS.MERGE merged_sketch 2 sketch_a sketch_b

29.5.5 HyperLogLog — Cardinality Estimation

HyperLogLog is built into Redis core (no module needed). It estimates the cardinality (unique count) of a set using only 12KB of memory, with a standard error of 0.81%:

PFADD unique_visitors "user:1" "user:2" "user:3"
PFCOUNT unique_visitors           # Returns: 3 (estimated)

# Merge: union across time windows
PFADD visitors_2024_01 "user:1" "user:4" "user:5"
PFADD visitors_2024_02 "user:2" "user:4" "user:6"
PFMERGE total_2024 visitors_2024_01 visitors_2024_02
PFCOUNT total_2024                # Returns: ~5 (4 unique estimated)

29.6 Production Guide

29.6.1 Deployment Options

# Docker: Redis Stack with RedisInsight UI
docker run -d \
  --name redis-stack \
  -p 6379:6379 \
  -p 8001:8001 \
  -v /local/data:/data \
  redis/redis-stack:latest

29.6.2 Memory Planning

Module indexes require significantly more memory than raw data:

29.6.3 Monitoring Module Health

# RedisSearch monitoring
FT.INFO product_idx
# num_docs: 152045
# num_records: 3041200
# inverted_sz_mb: 42.3
# indexing: 0          ← 0 = idle, 1 = background indexing

# Check index size and stats
FT._LIST              # List all indexes
FT.DEBUG DUMP_INVIDX product_idx redis  # Inspect posting list for term "redis"

# RedisTimeSeries monitoring
TS.INFO temperature
# totalSamples: 86400
# memoryUsage: 131072  ← bytes
# firstTimestamp: 1716000000000
# lastTimestamp: 1716086400000

# Module memory usage
MEMORY USAGE <module-managed-key>
INFO modules

29.7 Summary

Redis Modules transform Redis from a caching layer into a multi-purpose data platform:

• RedisSearch: Sub-millisecond full-text search and vector similarity queries entirely within Redis memory. Best when data fits in RAM and query latency must be below 1ms—a regime where Elasticsearch cannot compete.
• RedisJSON: Field-level JSON access with JSONPath. Eliminates the read-modify-write pattern for JSON documents and integrates natively with RedisSearch for secondary indexing.
• RedisTimeSeries: Optimized time series ingestion with 4–10x compression and built-in downsampling. Dramatically more memory-efficient than the naive Sorted Set approach for sensor data and metrics.
• RedisBloom: Probabilistic structures (Bloom filter, Cuckoo filter, Count-Min Sketch) that solve large-scale deduplication and frequency estimation problems with fixed, tiny memory footprints.

The trade-off is always memory amplification and operational complexity. Choose modules when you need Redis-level latency (sub-millisecond) and the dataset fits in memory; choose specialized external systems (Elasticsearch, InfluxDB, etc.) when data scale exceeds what RAM can hold.