Redis 7: Function, Sharded Pub/Sub and listpack

Chapter 41: Redis 7 New Features: Function, Sharded Pub/Sub, and listpack

Redis 7.0 is the most significant major release since Redis 6.0, delivering three architectural improvements: Function as a replacement for stateless Lua scripts, Sharded Pub/Sub to eliminate Cluster broadcast overhead, and listpack as the universal replacement for ziplist. This chapter analyzes the design motivation, implementation internals, and production considerations for each feature.

41.1 Function: Persistent Server-Side Scripts

41.1.1 Historical Limitations of Lua Scripting

Before Redis 7.0, the only way to extend server-side logic was EVAL with Lua scripts. The EVAL design is inherently "ephemeral":

Limitation 1: No Persistence Redis maintains a lua_scripts hash table keyed by SHA1 with script source as the value. SCRIPT LOAD caches a script into this table for later use via EVALSHA. However, this table is never written to RDB or AOF. After a Redis restart, the cache is empty and clients must re-execute SCRIPT LOAD. During rolling restarts or failovers, this easily causes NOSCRIPT errors in production.

Limitation 2: No Namespace All scripts share a flat SHA1 namespace. When multiple teams or applications share a Redis instance, script management becomes chaotic, with no way to clean up by ownership.

Limitation 3: No Version Management When a script changes, its SHA1 changes. Clients must synchronize the new SHA1 with each deployment, making smooth rolling updates impossible.

Limitation 4: Read-Only Replica Restrictions EVAL / EVALSHA are classified as write operations and cannot execute on read-only replicas, even when the script only performs reads. Redis 6.0 added EVAL_RO / EVALSHA_RO as a patch, but there was no systematic declaration mechanism.

41.1.2 Function Design Principles

Redis 7.0 introduces Functions with three core concepts:

• Library: A named container holding a group of functions; the unit of persistence and deployment.
• Function: A named callable unit within a library, invoked via FCALL.
• Engine: Currently only Lua (#!lua); the design supports additional languages in the future.

41.1.3 Loading and Calling Functions

# Load a library (REPLACE overwrites an existing library with the same name)
FUNCTION LOAD "#!lua name=mylib\n
local function my_hello(keys, args)\n
  return 'Hello ' .. args[1]\n
end\n
local function my_set(keys, args)\n
  return redis.call('SET', keys[1], args[1])\n
end\n
redis.register_function('hello', my_hello)\n
redis.register_function('myset', my_set)"

# Call a function (numkeys=0 means no key arguments)
FCALL hello 0 World
# → "Hello World"

# Call with a key argument
FCALL myset 1 mykey myvalue
# → OK

# Read-only variant: executable on replica nodes
FCALL_RO hello 0 World

The argument convention mirrors EVAL: FCALL funcname numkeys [key [key ...]] [arg [arg ...]]

41.1.4 Function Management Commands

# List all libraries and their functions
FUNCTION LIST
# → 1) 1) "library_name"
#      2) "mylib"
#   2) 1) "engine"
#      2) "LUA"
#   3) 1) "functions"
#      2) 1) 1) "name" 2) "hello"  3) "description" 4) nil  5) "flags" 6) (empty array)

# Show library source code
FUNCTION LIST WITHCODE

# Filter by library name
FUNCTION LIST LIBRARYNAME mylib

# Delete a library (removes all functions within it)
FUNCTION DELETE mylib

# Export all functions as binary payload (similar to DUMP)
FUNCTION DUMP
# → "\x\xf6\x00\xc3..."

# Restore from exported payload
FUNCTION RESTORE <dump-data>
FUNCTION RESTORE <dump-data> FLUSH   # clear existing functions before restore

# Remove all functions
FUNCTION FLUSH
FUNCTION FLUSH ASYNC   # non-blocking
FUNCTION FLUSH SYNC    # blocking

41.1.5 Persistence Mechanism

Functions are automatically persisted by Redis:

• RDB: The FUNCTION DUMP payload is stored as a special RDB type; automatically loaded on restart.
• AOF: FUNCTION LOAD commands are written to the AOF; replayed during recovery.
• Replication: FUNCTION LOAD propagates through the replication stream to all replicas; no extra action needed on replicas.

This completely eliminates the script cache loss problem that plagued Lua scripting.

41.1.6 Function Flags

Flags declared at registration time control how Redis handles the function:

-- no-writes: declares that the function performs no writes
-- allows execution on read-only replicas and under maxmemory pressure
redis.register_function{
  function_name = 'readonly_fn',
  callback = my_fn,
  flags = {'no-writes'}
}

-- allow-oom: permits execution even when maxmemory is exceeded
-- use with extreme caution
redis.register_function{
  function_name = 'critical_fn',
  callback = my_fn,
  flags = {'allow-oom'}
}

-- allow-stale: allows execution on a replica serving stale data
-- relevant when replica-serve-stale-data is set to no
redis.register_function{
  function_name = 'stale_ok',
  callback = my_fn,
  flags = {'allow-stale'}
}

41.1.7 Function vs EVAL Comparison

41.2 Sharded Pub/Sub (Redis 7.0)

41.2.1 The Problem with Traditional Pub/Sub in Cluster

In Redis Cluster, a traditional PUBLISH command broadcasts the message to every node in the cluster (all masters and replicas). Each node then forwards the message to any local clients subscribed to that channel.

Broadcast overhead analysis:

• Assume a cluster with 10 master nodes, publishing 100,000 messages per second
• Each message must be forwarded to 9 other nodes = 900,000 inter-node messages per second
• Nodes communicate via the Gossip protocol; heavy broadcasting consumes significant bandwidth and CPU

Root cause: Pub/Sub messages are not associated with any HashSlot, so they cannot leverage Cluster's slot-based routing. Every node must process every message.

41.2.2 Sharded Pub/Sub Design

Redis 7.0 introduces three new commands: SSUBSCRIBE, SUNSUBSCRIBE, and SPUBLISH. They route messages based on the HashSlot of the channel name.

Routing rules:

• Channel names are hashed using the same algorithm as keys (with {tag} support)
• SPUBLISH sends the message only to the node responsible for that slot (typically 1 master)
• That node notifies only its local subscribers and its own replicas

# Subscribe to a sharded channel (client library auto-routes to the correct node)
SSUBSCRIBE {user:1000}:events
# → 1) "ssubscribe"
#   2) "{user:1000}:events"
#   3) (integer) 1

# Unsubscribe
SUNSUBSCRIBE {user:1000}:events

# Publish to a sharded channel (goes only to the slot-owning node)
SPUBLISH {user:1000}:events "login"
# → (integer) 1   # number of subscribers that received the message (local node only)

# Inspect sharded channel state
PUBSUB SHARDCHANNELS             # list active sharded channels
PUBSUB SHARDNUMSUB channel1      # count subscribers for a sharded channel

41.2.3 Limitations and Considerations

1. No pattern subscription: SSUBSCRIBE only supports exact channel names; no wildcards.
2. Cluster-aware client required: The client library must understand Cluster topology to route SSUBSCRIBE / SPUBLISH to the correct node. Client support status:
   • Lettuce 6.2+: native support
   • Jedis 4.0+: use with JedisCluster
   • redis-py 4.3+: supported
3. Cluster-only feature: Meaningless in standalone or Sentinel mode where all slots are on one node.

41.2.4 Traditional vs Sharded Pub/Sub Comparison

41.3 listpack Replaces ziplist Universally

41.3.1 The ziplist Cascade Update Defect

ziplist is Redis's compact encoding for small containers. Each entry contains:

| prevlen (1 or 5 bytes) | encoding (1-9 bytes) | data |

prevlen stores the length of the previous entry for backward traversal. When the previous entry is shorter than 254 bytes, prevlen uses 1 byte; for 254 bytes or longer, it expands to 5 bytes.

Cascade update (CVE and performance issue): When inserting an entry in the middle of a ziplist causes a subsequent entry's prevlen to grow from 1 to 5 bytes (because the inserted entry is >= 254 bytes), the entry after that also needs its prevlen updated (since its predecessor just grew by 4 bytes). This can propagate through the entire list, producing O(N²) worst-case time complexity.

41.3.2 listpack Improvements

listpack (compact list) was introduced in Redis 5.0 for Stream entries and generalized in 7.0. Each entry format:

| encoding-type (1 byte) | data | backlen (1-5 bytes) |

The key change: backlen stores the length of the current entry itself (not the previous entry). For backward traversal, a reader uses the current entry's backlen to jump to its own head, then reads the prior entry. Inserting a new entry never modifies any subsequent entry, completely eliminating cascade updates.

Memory savings per entry:

• ziplist minimum: ~11 bytes (1 prevlen + 1 encoding + data + overhead)
• listpack minimum: ~7 bytes; saves 1–5 bytes per entry depending on prevlen size

41.3.3 Configuration Parameter Migration in Redis 7.0

# Hash
hash-max-listpack-entries 128    # formerly hash-max-ziplist-entries
hash-max-listpack-value 64       # formerly hash-max-ziplist-value

# ZSet
zset-max-listpack-entries 128    # formerly zset-max-ziplist-entries
zset-max-listpack-value 64       # formerly zset-max-ziplist-value

# List (7.0: small lists use listpack; large lists use quicklist of listpack nodes)
list-max-listpack-size -2        # formerly list-max-ziplist-size

Old parameter names remain backward-compatible but migrating to the new names is recommended.

Verifying encoding:

HSET myhash f1 v1
OBJECT ENCODING myhash
# → "listpack"   (before 7.0 this returned "ziplist")

# After exceeding the threshold, the encoding automatically upgrades
# (add 129 fields to myhash)
OBJECT ENCODING myhash
# → "hashtable"

41.4 Multi-Part AOF (Redis 7.0)

41.4.1 Problems with Traditional AOF Rewrite

The classic BGREWRITEAOF sequence:

1. Fork a child process; write the current memory state to a new file (appendonly.aof.tmp)
2. The parent process accumulates new write commands in the AOF rewrite buffer (in memory)
3. When the child finishes, the parent appends the rewrite buffer to the new AOF file
4. Atomic rename: appendonly.aof.tmp → appendonly.aof

Problem: Step 3 requires a lock on the main thread. If the write volume during rewrite is very high (buffer reaches hundreds of MB), this append-before-rename operation can block the main thread for hundreds of milliseconds.

41.4.2 Multi-Part AOF Architecture

Redis 7.0 splits the AOF into multiple files managed by a manifest:

/data/
  appendonlydir/
    appendonly.aof.1.base.rdb    # BASE file (full snapshot, RDB format)
    appendonly.aof.1.incr.aof    # Incremental AOF (round 1)
    appendonly.aof.2.incr.aof    # Incremental AOF (round 2, created after rewrite)
    appendonly.aof.manifest      # Manifest file

Manifest file format:

file appendonly.aof.1.base.rdb seq 1 type b
file appendonly.aof.1.incr.aof seq 1 type i
file appendonly.aof.2.incr.aof seq 2 type i

Improved rewrite sequence:

1. The parent creates appendonly.aof.2.incr.aof; all new writes go there immediately
2. A child process writes the current memory state as the new BASE file
3. When the child finishes, the manifest is atomically updated (new BASE, old INCRs can be deleted)
4. No buffer-append-to-large-file step; the parent has been writing real-time to the new INCR file throughout

At rewrite completion, the main thread only needs to update the tiny manifest file—eliminating the large-file append latency entirely.

41.4.3 Restart Recovery Sequence

1. Read appendonly.aof.manifest
2. Load BASE file (RDB format: direct load; AOF format: sequential replay)
3. Replay all INCR files in seq order
4. Recovery complete

Configuration (no changes needed; enabled by default in 7.0):

appendonly yes
appenddirname "appendonlydir"    # new in 7.0: specifies the AOF directory name
aof-use-rdb-preamble yes         # BASE file uses RDB format (better compression)

41.5 Additional Notable New Features

41.5.1 LMPOP / ZMPOP (Atomic Batch Pop)

# LMPOP: pop from the first non-empty list among multiple keys
# Syntax: LMPOP numkeys key [key ...] LEFT|RIGHT [COUNT count]
LMPOP 2 list1 list2 LEFT COUNT 5
# → 1) "list1"          # source key
#   2) 1) "elem1"
#      2) "elem2"

# ZMPOP: pop from the first non-empty sorted set among multiple keys
# Syntax: ZMPOP numkeys key [key ...] MIN|MAX [COUNT count]
ZMPOP 1 zset1 MIN COUNT 3
# → 1) "zset1"
#   2) 1) 1) "member1" 2) "1.0"
#      2) 1) "member2" 2) "2.0"
#      3) 1) "member3" 3) "3.0"

Use case: multi-queue priority consumers, draining tasks across multiple sorted sets atomically.

41.5.2 SINTERCARD (Set Intersection Cardinality)

# Returns only the count of intersection elements, not the elements themselves
# Avoids transferring large result sets when you only need the count
SINTERCARD 2 set1 set2
# → (integer) 42

# LIMIT: stop counting after N matches (performance optimization for large sets)
SINTERCARD 2 set1 set2 LIMIT 10
# → (integer) 10   (actual intersection may be larger; computation stops at 10)

41.5.3 XAUTOCLAIM (Stream Auto-Claim)

# Automatically claim PEL messages idle longer than min-idle-time milliseconds
# Transfers ownership to the specified consumer
# Syntax: XAUTOCLAIM key group consumer min-idle-time start [COUNT count] [JUSTID]
XAUTOCLAIM mystream mygroup consumer2 60000 0-0 COUNT 10
# → 1) "0-0"           # cursor for next call (next start ID)
#   2) 1) (list of transferred messages)
#   3) 1) (IDs that were in PEL but no longer exist in the stream — deleted entries)

Compared to XCLAIM, XAUTOCLAIM can scan and claim multiple timed-out messages in one call, making it the right tool for building consumer fault-recovery loops.

41.5.4 Upgrade Compatibility Checklist

Checklist when upgrading from Redis 6.x to 7.0:

# 1. Check for deprecated command usage
redis-cli COMMAND DOCS OBJECT   # review command docs for deprecated notices

# 2. Audit configuration parameter names (old names still work but update them)
grep -i ziplist /etc/redis/redis.conf   # find old parameter names

# 3. AOF directory migration (7.0 defaults to a directory, not a single file)
# appendonly.aof → appendonlydir/appendonly.aof.*
# Ensure the appenddirname path is writable before upgrading

# 4. Cluster: PUBLISH behavior is unchanged; only new SPUBLISH is sharded
# No changes needed to existing Pub/Sub code; migrate selectively as needed

Chapter Summary

Redis 7.0's core improvements each address a long-standing production pain point:

• Function gives server-side scripts persistence, version management, and namespacing, establishing a production-grade replacement for ad-hoc Lua EVAL usage.
• Sharded Pub/Sub drops Cluster's Pub/Sub propagation from O(N) broadcast to O(1) targeted delivery, solving the message fan-out bottleneck in large clusters.
• listpack eliminates the cascade-update defect of ziplist by removing the prevlen field, and becomes the default compact encoding for Hash, ZSet, and List in Redis 7.
• Multi-Part AOF architecturally eliminates the main-thread blocking window that occurred during traditional AOF rewrite, improving stability under high write workloads.

Together these features establish Redis 7 as a more robust foundation for production caching and storage systems.