MULTI/EXEC, Lua and Function

Chapter 27: MULTI/EXEC, Lua, and Function — Transactions and Scripting in Redis

27.1 The Problem: Multi-Step Atomicity

Redis single-threads command execution, so each individual command is atomic by definition. But real applications need something stronger: a sequence of read-then-write operations that cannot be interrupted. The canonical example—check balance, then debit—can go badly wrong if another client modifies the balance between the two operations.

Redis provides three solutions of increasing power:

1. MULTI/EXEC: Package multiple commands into an atomic batch
2. Lua scripting (EVAL): Run arbitrary server-side logic atomically
3. Functions (Redis 7+): Persisted, named Lua libraries

27.2 MULTI/EXEC Transactions

27.2.1 Basic Usage

# Start a transaction
MULTI
# Returns: OK

# Queue commands (server returns QUEUED, not the actual result)
SET balance 100
# Returns: QUEUED

DECRBY balance 30
# Returns: QUEUED

SET audit:log "deducted 30"
# Returns: QUEUED

# Execute all queued commands atomically
EXEC
# Returns:
# 1) OK
# 2) (integer) 70
# 3) OK

# Cancel a transaction in progress
MULTI
SET key value
DISCARD    # Flushes the queue and exits transaction mode

27.2.2 Internal Implementation

When the client sends MULTI, the server sets the CLIENT_MULTI flag on that connection. Subsequent commands are not executed immediately; instead they're appended to client->mstate.commands, an array of multiCmd structs:

/* server.h — transaction queue structures */
typedef struct multiCmd {
    robj **argv;                /* Command argument array */
    int argc;                   /* Argument count */
    struct redisCommand *cmd;   /* Pointer to command handler */
} multiCmd;

typedef struct multiState {
    multiCmd *commands;         /* Queued command array */
    int count;                  /* Queue depth */
    int cmd_flags;              /* OR'd flags from all queued commands */
    int minreplicas;            /* Required replicas before EXEC */
    time_t minreplicas_timeout;
} multiState;

When EXEC arrives, the main thread iterates mstate.commands and executes each command in sequence. No other client commands are interleaved. After completion, mstate is reset and the CLIENT_MULTI flag is cleared.

27.2.3 What "Atomic" Really Means in Redis

Redis transactions are NOT ACID transactions in the database sense:

Syntax errors vs. runtime errors behave differently:

# Syntax error — detected at QUEUED time
MULTI
SET key value
INCR key value extra_arg   # Wrong arity: caught immediately
EXEC
# Returns: EXECABORT — transaction aborted, NOTHING executed

# Runtime error — detected during EXEC
MULTI
SET mykey "hello"
INCR mykey               # Runtime error: can't INCR a string
SET another "world"
EXEC
# Returns:
# 1) OK           ← SET succeeded
# 2) WRONGTYPE    ← INCR failed, but execution continues
# 3) OK           ← SET succeeded

This is a deliberate design choice: Redis does not roll back on runtime errors. The application must handle partial failures explicitly.

27.2.4 WATCH — Optimistic Locking

WATCH extends MULTI/EXEC with optimistic concurrency control, implementing CAS (Check-And-Set) semantics:

WATCH balance               # Register optimistic lock on 'balance'

balance = GET balance       # Read current value outside the transaction
# balance = 100

MULTI
DECRBY balance 30           # QUEUED
EXEC
# If 'balance' was modified by anyone else between WATCH and EXEC:
#   Returns: nil (transaction aborted — must retry)
# Otherwise:
#   Returns: (integer) 70

How WATCH works internally:

/* server.h — watched key tracking */
/* server.watched_keys: global dict mapping key → list of watching clients */
/* client->watched_keys: set of keys this client is watching */

/* Called whenever any command modifies a key */
void signalModifiedKey(redisDb *db, robj *key) {
    touchWatchedKey(db, key);
    /* Marks all clients watching this key with CLIENT_DIRTY_CAS flag */
}

/* Checked at EXEC time */
if (c->flags & CLIENT_DIRTY_CAS) {
    execCommandAbortTransaction(c);  /* Returns nil to client */
    return;
}

Python pattern — CAS-based transfer with retry:

import redis
from redis.exceptions import WatchError

r = redis.Redis()

def atomic_transfer(from_key: str, to_key: str, amount: int, max_retries: int = 5) -> bool:
    with r.pipeline() as pipe:
        for attempt in range(max_retries):
            try:
                # Watch both keys before reading
                pipe.watch(from_key, to_key)

                # Read outside transaction (immediate mode)
                from_balance = int(pipe.get(from_key) or 0)
                to_balance = int(pipe.get(to_key) or 0)

                if from_balance < amount:
                    pipe.reset()
                    raise ValueError(f"Insufficient funds: have {from_balance}, need {amount}")

                # Open transaction window
                pipe.multi()
                pipe.decrby(from_key, amount)
                pipe.incrby(to_key, amount)
                pipe.execute()
                return True

            except WatchError:
                # Another client modified a watched key — retry
                if attempt == max_retries - 1:
                    raise RuntimeError(f"Transfer failed after {max_retries} attempts")
                continue

    return False

WATCH considerations:

• UNWATCH: explicitly release all watches (also happens automatically after EXEC or DISCARD)
• Client disconnect releases all watches automatically
• Watching many keys increases the cost of signalModifiedKey (linear scan)
• Under heavy contention, retries accumulate; consider switching to Lua for high-contention logic

27.3 Lua Scripting with EVAL

27.3.1 Why Lua Outperforms MULTI/EXEC

27.3.2 EVAL Syntax

EVAL script numkeys key [key ...] arg [arg ...]

• script: Lua source code as a string
• numkeys: how many of the following tokens are keys
• KEYS[i]: access key i inside the script (1-indexed)
• ARGV[i]: access arg i inside the script (1-indexed)

# Atomic bounded increment — won't exceed the cap
EVAL "
  local current = tonumber(redis.call('GET', KEYS[1])) or 0
  local cap = tonumber(ARGV[1])
  if current >= cap then
    return -1
  end
  return redis.call('INCR', KEYS[1])
" 1 request_counter 1000
# Returns new counter value, or -1 if cap reached

27.3.3 redis.call vs redis.pcall

-- redis.call: propagates errors to the caller immediately (stops script)
local val = redis.call('GET', KEYS[1])

-- redis.pcall: captures errors, script continues
local result = redis.pcall('HGET', KEYS[1], 'field')
if result.err then
    -- Handle gracefully
    return redis.error_reply("Expected hash, got wrong type")
end
return result

27.3.4 SCRIPT LOAD and EVALSHA — Saving Bandwidth

For long scripts used repeatedly, sending the full script body every call wastes bandwidth. Use SCRIPT LOAD to cache the script server-side and refer to it by SHA1 digest:

# Load script — returns SHA1 digest
SCRIPT LOAD "
  local current = tonumber(redis.call('GET', KEYS[1])) or 0
  if current >= tonumber(ARGV[1]) then return -1 end
  return redis.call('INCR', KEYS[1])
"
# Returns: "a42059b356c875f0717db19a51f6aaca9ae659ea"

# Execute by SHA1 — no script body transmitted
EVALSHA a42059b356c875f0717db19a51f6aaca9ae659ea 1 my_counter 100

# Check if a script is cached
SCRIPT EXISTS a42059b356c875f0717db19a51f6aaca9ae659ea
# Returns: 1 (cached) or 0 (not cached)

# Flush script cache (also cleared on server restart)
SCRIPT FLUSH

Client best practice: at application startup, SCRIPT LOAD all scripts and store the SHA1 digests. Use EVALSHA for all subsequent calls. On NOSCRIPT error (script not cached, e.g., after server restart), fall back to EVAL and re-cache.

27.3.5 Timeout Protection

# redis.conf
lua-time-limit 5000    # Max Lua execution time in milliseconds (default: 5000)

What happens when Lua times out:

1. Redis logs a warning
2. Redis begins accepting SCRIPT KILL and SHUTDOWN commands only
3. All other commands receive: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.
4. SCRIPT KILL terminates a safe script (one that hasn't executed any write commands)
5. If the script has already written data, only SHUTDOWN NOSAVE can save you (data loss!)

27.3.6 Production Lua Example — Token Bucket Rate Limiter

-- token_bucket.lua
-- KEYS[1]: rate limiter key
-- ARGV[1]: bucket capacity (max tokens)
-- ARGV[2]: refill rate (tokens per second)
-- ARGV[3]: current time in milliseconds
-- ARGV[4]: tokens requested by this call

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

-- Retrieve state, using defaults for first call
local last_tokens = tonumber(redis.call('HGET', key, 'tokens')) or capacity
local last_time = tonumber(redis.call('HGET', key, 'time')) or now

-- Calculate token refill since last call
local elapsed_ms = math.max(0, now - last_time)
local refilled = elapsed_ms * refill_rate / 1000.0
local tokens = math.min(capacity, last_tokens + refilled)

-- Decide whether to allow the request
local allowed = 0
if tokens >= requested then
    tokens = tokens - requested
    allowed = 1
end

-- Save state with a safety TTL
redis.call('HSET', key, 'tokens', tokens, 'time', now)
redis.call('EXPIRE', key, 3600)

return { allowed, math.floor(tokens) }

# Python client usage
import time, redis

r = redis.Redis()
sha = r.script_load(open('token_bucket.lua').read())

def is_allowed(user_id: str, capacity: int = 100, rate: int = 10) -> bool:
    key = f"ratelimit:{user_id}"
    now_ms = int(time.time() * 1000)
    result = r.evalsha(sha, 1, key, capacity, rate, now_ms, 1)
    allowed, remaining = result
    return allowed == 1

27.4 Redis 7 Functions

27.4.1 What's Wrong with Lua Scripts

27.4.2 Loading and Calling Functions

# Load a function library (Lua engine)
FUNCTION LOAD "#!lua name=banking
redis.register_function('get_with_default', function(keys, args)
  local val = redis.call('GET', keys[1])
  if val == false then
    return args[1]   -- return default value when key missing
  end
  return val
end)
redis.register_function('atomic_transfer', function(keys, args)
  local amount = tonumber(args[1])
  local from_bal = tonumber(redis.call('GET', keys[1])) or 0
  if from_bal < amount then
    return redis.error_reply('ERR insufficient balance')
  end
  redis.call('DECRBY', keys[1], amount)
  redis.call('INCRBY', keys[2], amount)
  return amount
end)"

# Call a function
FCALL get_with_default 1 user:1000:score 0
FCALL atomic_transfer 2 account:A account:B 50

# Call a read-only function (safe to run on replicas)
FCALL_RO get_with_default 1 user:1000:score 0

27.4.3 Function Management Commands

# List all loaded function libraries
FUNCTION LIST

# List with code content
FUNCTION LIST WITHCODE

# Filter by library name
FUNCTION LIST LIBRARYNAME banking

# Get execution statistics
FUNCTION STATS

# Delete a library
FUNCTION DELETE banking

# Upgrade: replace existing library
FUNCTION LOAD REPLACE "#!lua name=banking
redis.register_function(...)
"

# Migrate between servers
src_dump=$(redis-cli FUNCTION DUMP)
redis-cli -h new-server FUNCTION RESTORE "$src_dump"

27.4.4 Persistence Integration

Functions are embedded in RDB snapshots and replayed from AOF:

RDB file structure (Redis 7+):
┌─────────────────────────────────────┐
│  RDB header                         │
│  DB 0 key-value data                │
│  ...                                │
│  FUNCTION library: banking          │  ← Functions stored here
│    function: get_with_default       │
│    function: atomic_transfer        │
│  EOF + checksum                     │
└─────────────────────────────────────┘

AOF records FUNCTION LOAD commands, so functions are also replayed on AOF-based recovery. This eliminates the "reconnect and reload scripts on startup" boilerplate that Lua scripts required.

27.5 Performance and Decision Guide

27.5.1 Round Trip and Atomicity Comparison

27.5.2 Decision Tree

Need multi-step atomicity?
├── No  → Single command or Pipeline
└── Yes
    ├── Simple sequence, no conditional logic?
    │   └── MULTI/EXEC
    ├── Need to read then conditionally write?
    │   ├── Low contention → WATCH + MULTI/EXEC
    │   └── High contention or complex logic → EVAL or Function
    └── Need logic to survive server restarts?
        └── Redis 7 Function

27.5.3 Common Mistakes

Mistake 1: Treating MULTI/EXEC as a full database transaction

MULTI/EXEC does not roll back. If command 2 of 5 fails at runtime, commands 1 and 3–5 still execute. Applications must handle partial failure with compensating logic or switch to Lua.

Mistake 2: Writing long Lua scripts that block Redis

Lua execution is uninterruptible. A script that runs for 100ms blocks all other clients for 100ms. Keep individual Lua executions under 5ms; for complex workflows, break them into smaller atomic steps.

Mistake 3: Using KEYS inside EVAL in Cluster mode

Cluster routes commands by key slot. Inside EVAL, only keys that share the same slot (same node) can be accessed. Use hash tags {} to force co-location:

# WRONG: key1 and key2 might be on different nodes
EVAL "redis.call('SET', KEYS[1], redis.call('GET', KEYS[2]))" 2 key1 key2

# CORRECT: hash tag forces both to the same slot
EVAL "redis.call('SET', KEYS[1], redis.call('GET', KEYS[2]))" 2 {user:1}:dest {user:1}:src

Mistake 4: Using math.random() in Lua for data that is replicated

math.random() is seeded differently on primary and replicas, causing divergence. Use redis.call('TIME') for time-based randomness, or generate random values on the client side.

27.6 Production Operations

27.6.1 Diagnosing BUSY Errors

# Check if a script is running
redis-cli INFO server | grep lua_scripts

# Kill a safe (read-only) script
SCRIPT KILL

# Last resort — if script has written data, only SHUTDOWN works
# DATA LOSS warning!
SHUTDOWN NOSAVE

# Prevention: always set lua-time-limit conservatively
# and have monitoring alert on BUSY errors

27.6.2 Monitoring

# Slow log catches slow Lua executions
SLOWLOG GET 10

# Real-time EVAL monitoring (use sparingly in production)
redis-cli monitor | grep -E 'EVAL|EVALSHA|FCALL'

# Server stats
INFO stats | grep lua
# lua_scripts_count: N

27.6.3 Using the Lua Debugger

# Interactive Lua debugger (Redis 3.2+)
redis-cli --ldb --eval my_script.lua key1 key2 , arg1 arg2

# Debugger commands
# s       — step
# n       — next (step over)
# c       — continue
# l       — list source
# b 10    — breakpoint at line 10
# p expr  — print expression
# redis   — inspect Redis state

27.7 Summary

Redis offers a progression of atomic operation capabilities:

1. MULTI/EXEC: Simple, low-overhead batching for sequences without branching. Useful but semantically weaker than true transactions—no rollback.
2. WATCH: Adds optimistic locking to MULTI/EXEC, enabling safe CAS patterns. Works well under low-to-medium contention.
3. Lua EVAL: Full server-side computation, truly uninterruptible, one network round trip. The right tool for complex conditional logic, but scripts aren't persisted.
4. Function (Redis 7): Everything Lua offers, plus persistence in RDB, named libraries, and lifecycle management. The forward-looking choice for production systems.

For new projects on Redis 7+, Functions should be the default choice for any non-trivial server-side logic. Existing EVAL scripts can coexist and be migrated to Functions incrementally.