redisObject: 12 Encodings and Memory Layout
Chapter 4: The redisObject System: 12 Encodings and Memory Layout
4.1 Why redisObject Exists
Every value stored in Redis — whether a string from SET, a list element from RPUSH, or a member of a sorted set — is represented internally as a redisObject (typedef robj). This unified wrapper serves three functions:
- Type information: What Redis type does this value belong to? (
OBJ_STRING,OBJ_LIST, etc.) - Encoding selection: What underlying data structure is used? (e.g., a small hash might use listpack, a large one uses a hash table)
- Memory management: Reference counting, LRU/LFU metadata
The encoding layer is the key insight: the same logical type can use different physical representations depending on the data's characteristics. A HASH with 10 small fields uses a compact listpack; the same hash with 200 fields uses a full hash table. The selection is automatic and transparent to clients.
4.2 The robj Structure: Complete Field Analysis
Source file: src/server.h
typedef struct redisObject {
unsigned type:4; /* Object type: OBJ_STRING/LIST/SET/ZSET/HASH */
unsigned encoding:4; /* Encoding: one of 12 OBJ_ENCODING_* values */
unsigned lru:LRU_BITS; /* LRU_BITS=24: LRU timestamp OR LFU counter */
int refcount; /* Reference count; shared objects use INT_MAX */
void *ptr; /* Pointer to underlying data structure */
} robj;
4.2.1 Memory Layout (64-bit system)
Byte offset Field Width Notes
────────────────────────────────────────────────────────────
0 [type:4] } These three bit fields share
0 [encoding:4] } 4 bytes one 32-bit unsigned int
0 [lru:24] }
4 refcount 4 bytes int (signed 32-bit)
8 ptr 8 bytes 64-bit pointer
────────────────────────────────────────────────────────────
Total: 16 bytes (excluding the data pointed to by ptr)
The type, encoding, and lru fields are C bit fields packed into a single uint32_t. On a 64-bit system, three separate fields would consume at least 12 bytes (3 × 4-byte int minimum); bit fields compress them to 4 bytes, saving 8 bytes per object.
Verification with DEBUG OBJECT:
SET testkey "hello"
DEBUG OBJECT testkey
# Value at:0x7f8c2a0b1340 refcount:1 encoding:embstr serializedlength:5
# lru:16234567 lru_seconds_idle:2 type:string
OBJECT ENCODING testkey # → "embstr"
OBJECT REFCOUNT testkey # → 1
OBJECT IDLETIME testkey # → seconds since last access
OBJECT FREQ testkey # → LFU frequency (if maxmemory-policy is LFU-based)
MEMORY USAGE testkey # → bytes consumed (including pointers, structs)
4.2.2 The type Field (4 bits)
/* src/server.h */
#define OBJ_STRING 0 /* String value */
#define OBJ_LIST 1 /* List */
#define OBJ_SET 2 /* Set */
#define OBJ_ZSET 3 /* Sorted set */
#define OBJ_HASH 4 /* Hash */
#define OBJ_MODULE 5 /* Module-managed type */
#define OBJ_STREAM 6 /* Stream (Redis 5.0+) */
The TYPE key command reads this field and maps it to the human-readable string ("string", "list", "set", "zset", "hash", "stream").
4.2.3 The encoding Field (4 bits): 12 Encodings
#define OBJ_ENCODING_RAW 0 /* SDS string (> 44 bytes) */
#define OBJ_ENCODING_INT 1 /* Integer (stored in ptr directly) */
#define OBJ_ENCODING_HT 2 /* Hash table (dict) */
#define OBJ_ENCODING_ZIPMAP 3 /* Deprecated zipmap (unused since 2.6) */
#define OBJ_ENCODING_LINKEDLIST 4 /* Deprecated linked list (removed in 7.0) */
#define OBJ_ENCODING_ZIPLIST 5 /* Compact list (replaced by listpack in 7.0) */
#define OBJ_ENCODING_INTSET 6 /* Integer set (sorted array) */
#define OBJ_ENCODING_SKIPLIST 7 /* Skip list + hash table (large ZSet) */
#define OBJ_ENCODING_EMBSTR 8 /* Embedded SDS (<= 44 bytes, contiguous) */
#define OBJ_ENCODING_QUICKLIST 9 /* Linked list of listpack nodes (List) */
#define OBJ_ENCODING_STREAM 10 /* Stream (radix tree + listpack) */
#define OBJ_ENCODING_LISTPACK 11 /* Compact sequence (small Hash/List/ZSet) */
4.3 Deep Dive: Each Encoding
4.3.1 INT Encoding: Integer Stored in the Pointer
When a String value is representable as a long integer (within LLONG_MIN to LLONG_MAX), Redis avoids allocating an SDS entirely. The integer value is cast to a pointer and stored directly in robj.ptr:
/* src/object.c */
robj *createStringObjectFromLongLong(long long value) {
robj *o;
if (value >= 0 && value < OBJ_SHARED_INTEGERS) {
/* Use shared integer object */
o = shared.integers[value];
} else {
o = createObject(OBJ_STRING, NULL);
o->encoding = OBJ_ENCODING_INT;
o->ptr = (void *)((long)value); /* Store integer in ptr field */
}
return o;
}
The shared integer pool: For values 0–9999, Redis pre-creates 10,000 objects at startup:
/* src/server.h */
#define OBJ_SHARED_INTEGERS 10000
/* src/object.c — called during initServer() */
void createSharedObjects(void) {
int j;
for (j = 0; j < OBJ_SHARED_INTEGERS; j++) {
shared.integers[j] = makeObjectShared(
createObject(OBJ_STRING, (void*)(long)j)
);
shared.integers[j]->encoding = OBJ_ENCODING_INT;
}
}
When SET key 42 is executed, the value object is shared.integers[42] — no allocation, just an incrRefCount.
Why only 0–9999: The choice balances startup cost and memory footprint against coverage. Values 0–9999 cover the vast majority of practical integers: HTTP status codes, database IDs, counters, ages, quantities, boolean-like flags. Extending the pool to 100,000 would consume 1.6MB of RAM for the objects themselves and add measurable startup latency.
Verifying shared objects:
SET a 42
SET b 42
OBJECT REFCOUNT a # → 2147483647 (INT_MAX = shared object marker)
OBJECT REFCOUNT b # → 2147483647
SET c 10001 # Outside shared pool
OBJECT REFCOUNT c # → 1 (independently allocated)
4.3.2 EMBSTR Encoding: Contiguous Allocation for Short Strings
EMBSTR is Redis's most elegant memory optimization. For strings of 44 bytes or fewer, the robj and the sdshdr8 header are allocated in a single contiguous malloc() call:
Single malloc (64 bytes from jemalloc):
┌──────────────────────────────────────────────────────────────────────────┐
│ robj (16 bytes) │ sdshdr8 (3 bytes) │ buf (up to 44 + '\0') │
│ ┌──────────────────────┐ │ ┌───────────────┐ │ 'h''e''l''l''o''\0' │
│ │type:4 encoding:4 │ │ │len: uint8_t │ │ │
│ │lru:24 │ │ │alloc:uint8_t │ │ │
│ │refcount: (int) │ │ │flags:uint8_t │ │ │
│ │ptr: ─────────────────┼───┼─┼──────────────→│ │ │
│ └──────────────────────┘ │ └───────────────┘ │ │
└──────────────────────────────────────────────────────────────────────────┘
The 44-byte threshold derivation:
jemalloc size class: 64 bytes (smallest class that fits the structure)
Available budget: 64 bytes
- robj overhead: -16 bytes
- sdshdr8 header: -3 bytes (len + alloc + flags, each uint8_t)
- NUL terminator (\0): -1 byte
─────────────────────────────────
Maximum string content: 44 bytes
If Redis used sdshdr16 for embstr (which would be needed for strings over 255 bytes, but isn't), the header would be 5 bytes and the threshold would be 42 bytes. The 44-byte limit is specifically tied to sdshdr8's 3-byte header and jemalloc's 64-byte size class.
Why EMBSTR is effectively immutable:
Any in-place modification of an embstr would require reallocating the combined robj+sdshdr block (since sdsMakeRoomFor may need to move the allocation to grow the string). Redis takes the simpler path: any modification converts embstr to raw first:
/* src/object.c */
robj *dbUnshareStringValue(redisDb *db, robj *key, robj *o) {
if (o->encoding == OBJ_ENCODING_EMBSTR) {
robj *decoded = getDecodedObject(o);
o = dbUnshareStringValue(db, key, decoded);
/* decoded is now OBJ_ENCODING_RAW */
}
// ...
}
SET mykey "hello"
OBJECT ENCODING mykey # → embstr
APPEND mykey " world" # Triggers embstr → raw conversion
OBJECT ENCODING mykey # → raw (even though "hello world" is only 11 bytes!)
# Once converted to raw, never converted back to embstr
Performance advantage of embstr: Because the robj and SDS are contiguous, a single pointer dereference reaches both the object metadata and the string data. On modern CPUs with 64-byte cache lines, the entire structure fits in one cache line — zero cache misses after the initial load.
4.3.3 RAW Encoding: Separately Allocated SDS
For strings longer than 44 bytes, the robj and SDS are allocated independently:
robj (16 bytes, malloc #1) SDS (separate malloc #2)
┌──────────────────────────┐ ┌──────────────────────────────────┐
│ type: OBJ_STRING │ │ sdshdr32: │
│ encoding: RAW │ │ len: uint32_t (4 bytes) │
│ lru: ... │ │ alloc: uint32_t (4 bytes) │
│ refcount: 1 │ │ flags: uint8_t (1 byte) │
│ ptr: ────────────────────┼────→│ buf[]: string data + '\0' │
└──────────────────────────┘ └──────────────────────────────────┘
Two cache misses required: one for robj, one for the SDS data.
sdshdr32 is used when string length > 65535 bytes (uint16_t overflow). sdshdr64 kicks in above ~4 GB (uint32_t overflow). In practice, Redis values should never approach these sizes in production.
4.3.4 LISTPACK Encoding: Compact Binary Sequence
Listpack (introduced in Redis 7.0) replaced ziplist as the compact encoding for small Hash, List, and ZSet values. The key improvement over ziplist: listpack entries do not store the length of the previous entry, eliminating the cascade update problem.
Binary format:
Listpack layout:
┌────────────┬──────────────┬─────────────────────────────┬──────┐
│ Total Bytes│ Num Elements │ Entry_0 | Entry_1 | Entry_N │ 0xFF │
│ (4 bytes) │ (2 bytes) │ (variable length) │ End │
└────────────┴──────────────┴─────────────────────────────┴──────┘
Entry structure:
┌──────────────────────────┬─────────────────────────────┐
│ Encoding + Data (var len)│ Backlen (1–5 bytes) │
└──────────────────────────┴─────────────────────────────┘
Encoding byte for small unsigned int (7-bit value, 0–127):
0xxxxxxx — single byte, value = xxxxxxx
Encoding for medium unsigned int (13-bit value):
110xxxxx yyyyyyyy — 13-bit value split across 2 bytes
Encoding for string:
101xxxxx [len-1 bytes] [string bytes] — string up to 4095 bytes
Ziplist cascade update problem (why listpack fixes it):
In ziplist, each entry stored prevlen (length of previous entry in 1 or 5 bytes). If entry N's size changed from < 254 bytes to ≥ 254 bytes, its prevlen field in entry N+1 needed to grow from 1 to 5 bytes — which changed entry N+1's size, which might change entry N+2's prevlen, and so on. Worst case: O(n) updates for a single insertion. Listpack eliminates prevlen, so no cascade is possible.
4.3.5 QUICKLIST Encoding: Linked List of Listpacks
Large Redis Lists use quicklist: a doubly-linked list where each node contains a listpack:
typedef struct quicklist {
quicklistNode *head; /* First node */
quicklistNode *tail; /* Last node */
unsigned long count; /* Total number of elements */
unsigned long len; /* Number of quicklistNodes */
signed int fill : QL_FILL_BITS; /* fill factor for individual nodes */
unsigned int compress : QL_COMP_BITS; /* depth of end nodes not compressed */
unsigned int bookmark_count: QL_BM_BITS;
quicklistBookmark bookmarks[];
} quicklist;
typedef struct quicklistNode {
struct quicklistNode *prev;
struct quicklistNode *next;
unsigned char *entry; /* listpack or raw string (if LZF compressed) */
size_t sz; /* entry size in bytes */
unsigned int count : 16; /* count of items in listpack */
unsigned int encoding : 2; /* RAW or LZF compressed */
unsigned int container : 2; /* PLAIN or PACKED */
unsigned int recompress : 1;
unsigned int attempted_compress : 1;
unsigned int dont_compress : 1;
unsigned int extra : 9;
} quicklistNode;
Configuration:
list-max-listpack-size 128 # Positive: max elements per listpack node
list-max-listpack-size -2 # Negative: max bytes per node
# -1 = 4KB, -2 = 8KB, -3 = 16KB, -4 = 32KB, -5 = 64KB
list-compress-depth 0 # How many end nodes to leave uncompressed
# 0 = no compression, 1 = compress all except head+tail
# 2 = don't compress head, head->next, tail->prev, tail
4.3.6 HT Encoding: Full Hash Table
When a Hash exceeds hash-max-listpack-entries (default 128) or hash-max-listpack-value (default 64 bytes), it upgrades to HT (hash table encoding backed by Redis's dict):
/* src/dict.h */
typedef struct dict {
dictType *type;
dictht ht[2]; /* Two hash tables: ht[0] is active, ht[1] used during rehash */
long rehashidx; /* Rehash position: -1 = not rehashing */
unsigned pauserehash; /* If > 0, rehash is paused */
signed char pauseAutoResize;
} dict;
typedef struct dictht {
dictEntry **table; /* Array of bucket pointers */
unsigned long size; /* Number of buckets (always power of 2) */
unsigned long sizemask;/* size - 1, used for masking hash value */
unsigned long used; /* Number of entries in the table */
} dictht;
Incremental rehashing (the mechanism that prevents blocking on resize):
Initial state: ht[0] has 1024 buckets, load factor > 1 → trigger resize
Action: allocate ht[1] with 2048 buckets, set rehashidx = 0
On every HGET, HSET, or HDEL call while rehashidx >= 0:
1. Migrate ht[0].table[rehashidx] (one bucket's chain) to ht[1]
2. Increment rehashidx
3. Execute the user's command normally
Lookup during rehash:
1. Check ht[0] first
2. If not found, check ht[1]
(ensures correctness during migration)
Writes during rehash:
Always write to ht[1] (prevents writing to the old table)
Completion:
When rehashidx == ht[0].size → free ht[0], set ht[0] = ht[1], ht[1] = empty
Reset rehashidx = -1
The incremental approach means no single operation takes more than O(1) extra time for rehashing, at the cost of slightly more memory during the migration window (both tables exist simultaneously).
4.3.7 SKIPLIST Encoding: Skip List + Hash Table
Large ZSet values use a dual-structure encoding combining a skip list (for range operations) and a hash table (for O(1) score lookup):
typedef struct zset {
dict *dict; /* member → score: O(1) score lookup */
zskiplist *zsl; /* score-ordered structure: O(log N) range queries */
} zset;
Why maintain both structures simultaneously?
| Operation | Uses | Complexity |
|---|---|---|
ZSCORE key member |
dict only | O(1) |
ZRANK key member |
skip list | O(log N) |
ZRANGE key 0 -1 |
skip list | O(log N + M) |
ZADD key score member |
both (insert to skip list + dict) | O(log N) |
ZREM key member |
both (delete from both) | O(log N) |
Maintaining both costs extra memory (each member is stored twice — once in dict, once in skip list node), but it delivers O(1) score access without needing to traverse the skip list.
Skip list structure:
Level 4: ─────────────────────────────────────────── [50:nodeE] ──→ NULL
Level 3: ─────────── [10:nodeB] ───────────────────── [50:nodeE] ──→ NULL
Level 2: ─── [5:A] ── [10:B] ── [20:C] ──────────── [50:E] ──────→ NULL
Level 1: ─── [5:A] ── [10:B] ── [20:C] ── [35:D] ── [50:E] ──────→ NULL
Each node:
- score (double)
- member (SDS string)
- backward pointer (for reverse traversal)
- forward pointers at each level (span = how many level-0 nodes this pointer skips)
Node level assigned randomly on insertion: P(level = k) = 0.25^(k-1) × 0.75
Expected height: O(log N), max height: ZSKIPLIST_MAXLEVEL (32)
The span of each forward pointer enables O(log N) rank computation without a separate rank tracking structure.
4.3.8 INTSET Encoding: Sorted Integer Array
When a Set contains only integers and the count is ≤ set-max-intset-entries (default 512), Redis uses intset — a sorted array of integers:
/* src/intset.h */
typedef struct intset {
uint32_t encoding; /* INTSET_ENC_INT16, INT32, or INT64 */
uint32_t length; /* Number of elements */
int8_t contents[]; /* Sorted integer array (actual element size per encoding) */
} intset;
Encoding upgrade: intset uses the smallest integer type that can hold all elements:
- All elements fit in int16_t (−32768 to 32767) → 2 bytes per element
- Any element exceeds int16_t range → upgrade to int32_t (4 bytes/element), copy entire array
- Any element exceeds int32_t range → upgrade to int64_t (8 bytes/element), copy entire array
Binary search for lookup: O(log N). Insertion maintains sorted order: O(N) for the shift operation, but since maximum size is 512 elements, this is at most 512 × 8 = 4096 bytes copied — negligible.
SADD intset 1 2 3 4 5
OBJECT ENCODING intset # → intset
SADD intset "hello" # Non-integer triggers upgrade to HT
OBJECT ENCODING intset # → hashtable (no downgrade back to intset)
4.4 Encoding Transitions: Thresholds and Rules
4.4.1 String
Integer representable as long → INT
Length ≤ 44 bytes → EMBSTR
Length > 44 bytes → RAW
Any modification to EMBSTR → RAW (no downgrade back to EMBSTR)
4.4.2 Hash
Initial (small) → LISTPACK
Trigger upgrade to HT when:
count > hash-max-listpack-entries (default 128) OR
any field OR value length > hash-max-listpack-value (default 64 bytes)
HT never downgrades to LISTPACK (even after deletions)
# Demonstrating upgrade
HSET h f1 v1
OBJECT ENCODING h # → listpack
HSET h f2 $(python3 -c "print('x'*65)") # 65-byte value
OBJECT ENCODING h # → hashtable
HDEL h f2
OBJECT ENCODING h # → hashtable (no downgrade)
4.4.3 List
Initial (small) → LISTPACK
Trigger upgrade to QUICKLIST when:
count > list-max-listpack-size (positive mode: default 128) OR
any element byte size exceeds node limit (negative mode)
4.4.4 Set
All integers AND count ≤ set-max-intset-entries (512) → INTSET
Non-integer element added OR count exceeds threshold → HT
No downgrade from HT to INTSET
4.4.5 ZSet
Initial (small) → LISTPACK
Trigger upgrade to SKIPLIST when:
count > zset-max-listpack-entries (default 128) OR
any member length > zset-max-listpack-value (default 64 bytes)
No downgrade from SKIPLIST to LISTPACK
4.5 The 24-bit LRU/LFU Field: Dual Use
The lru bit field in robj (24 bits) serves radically different purposes depending on the configured maxmemory-policy.
4.5.1 LRU Mode
#define LRU_CLOCK_MAX ((1<<24)-1) /* 16,777,215 — wraps at ~194 days */
#define LRU_CLOCK_RESOLUTION 1000 /* Update resolution: 1000ms = 1 second */
unsigned int getLRUClock(void) {
return (mstime() / LRU_CLOCK_RESOLUTION) & LRU_CLOCK_MAX;
}
Redis uses approximate LRU — not exact. On each eviction decision, Redis samples maxmemory-samples (default 5) keys at random and evicts the one with the oldest LRU timestamp. Setting maxmemory-samples 10 gives closer-to-true-LRU behavior at the cost of more CPU per eviction.
The 24-bit timestamp wraps at 16,777,215 seconds ≈ 194 days. Keys idle for longer than 194 days have their LRU timestamp appear recent after wrapping, but this is practically irrelevant — such keys would have been evicted long before.
4.5.2 LFU Mode
In LFU-based policies (allkeys-lfu, volatile-lfu), the 24-bit field splits:
Bit 23 (MSB) ............. Bit 8 | Bit 7 .......... Bit 0 (LSB)
─────────────────────────────────────────────────────────────────────
16 bits: LDT (Last Decrement Time) | 8 bits: Morris counter
minute-resolution timestamp | logarithmic access frequency
Morris counter logarithmic increment:
uint8_t LFULogIncr(uint8_t counter) {
if (counter == 255) return 255; /* Saturate at maximum */
double r = (double)rand() / RAND_MAX;
double baseval = counter - LFU_INIT_VAL;
if (baseval < 0) baseval = 0;
double p = 1.0 / (baseval * server.lfu_log_factor + 1);
if (r < p) counter++; /* Probabilistic increment */
return counter;
}
Approximate counter values (with default lfu_log_factor = 10):
| Counter value | Approximate access count |
|---|---|
| 1 | 1 |
| 10 | ~10 |
| 50 | ~1,000 |
| 100 | ~10,000 |
| 200 | ~1,000,000 |
| 255 (max) | > 10,000,000 |
Decay mechanism: The 16-bit LDT (Last Decrement Time, minute-resolution) tracks when the counter was last decremented. On each access, lfu_decay_time (default: 1 minute per unit) is applied:
unsigned long LFUDecrAndReturn(robj *o) {
unsigned long ldt = o->lru >> 8; /* Extract high 16 bits */
unsigned long counter = o->lru & 255; /* Extract low 8 bits */
unsigned long num_periods = LFUTimeElapsed(ldt) / server.lfu_decay_time;
if (num_periods) {
counter = (num_periods > counter) ? 0 : counter - num_periods;
}
return counter;
}
This means a key accessed millions of times but not touched for 10 minutes drops by 10 units — hot keys retain their high counter, cold keys decay quickly.
4.6 Reference Counting
#define OBJ_SHARED_REFCOUNT INT_MAX /* 2147483647 — immutable shared objects */
#define OBJ_STATIC_REFCOUNT (INT_MAX-1)/* stack-allocated temporary objects */
#define OBJ_FIRST_SPECIAL_REFCOUNT OBJ_STATIC_REFCOUNT
void incrRefCount(robj *o) {
if (o->refcount < OBJ_FIRST_SPECIAL_REFCOUNT) o->refcount++;
}
void decrRefCount(robj *o) {
if (o->refcount == 1) {
switch(o->type) {
case OBJ_STRING: freeStringObject(o); break;
case OBJ_LIST: freeListObject(o); break;
case OBJ_SET: freeSetObject(o); break;
case OBJ_ZSET: freeZsetObject(o); break;
case OBJ_HASH: freeHashObject(o); break;
/* ... */
}
zfree(o); /* Free the robj struct itself */
} else {
o->refcount--;
}
}
Shared object refcount: The 10,000 shared integer objects and a set of commonly used string objects (empty string, "OK", error messages) have refcount = INT_MAX. This prevents them from ever being freed by decrRefCount — they are permanent fixtures of the server state.
Checking refcount in production:
SET mykey 42
OBJECT REFCOUNT mykey # → 2147483647 (shared integer 42)
SET mykey 99999 # Outside shared pool
OBJECT REFCOUNT mykey # → 1
# After OBJECT REFCOUNT on a key referenced in multiple structures,
# refcount reflects the number of data structure references
4.7 Real Memory Consumption Example
Case: SET user:1001 "Alice" — how many bytes does the value object consume?
String "Alice" (5 bytes, EMBSTR encoding):
robj portion:
[type:4 + encoding:4 + lru:24] = 4 bytes
refcount = 4 bytes
ptr = 8 bytes
Subtotal: = 16 bytes
sdshdr8 portion (packed into same allocation):
len (uint8_t) = 1 byte
alloc (uint8_t) = 1 byte
flags (uint8_t) = 1 byte
Subtotal: = 3 bytes
buf[] portion:
'A','l','i','c','e','\0' = 6 bytes
Raw sum: 16 + 3 + 6 = 25 bytes
jemalloc size class: 32 bytes (rounds up to nearest power of two)
Actual allocation: 32 bytes for value object
Full key-value pair cost:
dictEntry (the hash table entry):
key pointer: 8 bytes
value pointer: 8 bytes
next pointer: 8 bytes
Subtotal: 24 bytes
Key "user:1001" as robj (9 bytes, EMBSTR):
robj + sdshdr8 + buf: 16 + 3 + 9 + 1 = 29 bytes → 32 bytes allocated
Value "Alice" as robj (5 bytes, EMBSTR):
32 bytes (calculated above)
Total for one KV pair: 24 + 32 + 32 = 88 bytes
Verify with MEMORY USAGE:
SET user:1001 "Alice"
MEMORY USAGE user:1001 # Returns bytes including key, value, and overhead
# Typical output: 56 (the exact number depends on Redis version and jemalloc behavior)
The discrepancy from our calculation is because MEMORY USAGE uses zmalloc_size() which reports the actual jemalloc-allocated size for each pointer, and includes some Redis-internal overhead per key in the main dictionary.
4.8 Practical Tuning Implications
Understanding encoding thresholds enables memory/CPU trade-offs:
Memory optimization (use more CPU for better memory density):
# Keep more hashes in compact listpack encoding
CONFIG SET hash-max-listpack-entries 256 # default: 128
CONFIG SET hash-max-listpack-value 128 # default: 64 bytes
# Similarly for other types
CONFIG SET zset-max-listpack-entries 256
CONFIG SET set-max-intset-entries 1024
CPU optimization (accept higher memory use for faster operations):
# Smaller thresholds → earlier upgrade to hash table → faster O(1) operations
CONFIG SET hash-max-listpack-entries 32 # Upgrade to HT sooner
CONFIG SET zset-max-listpack-entries 64
Monitoring encoding distribution in a running cluster:
redis-cli --bigkeys # Identifies largest keys by type
redis-cli object encoding mykey # Encoding of specific key
# For aggregate statistics, use DEBUG RELOAD + INFO memory
# or third-party tools like redis-memory-analyzer
4.9 Summary
The robj encoding system is the mechanism that lets Redis be simultaneously memory-efficient for small datasets and fast for large ones. The 16-byte robj wrapper, combined with 12 different underlying encodings and automatic promotion/demotion rules, creates a self-tuning storage layer that adapts to data characteristics without developer intervention.
The key practical takeaways:
- Strings under 44 bytes get the cache-friendly EMBSTR treatment (single cache line, one malloc)
- Small collections stay in listpack (compact binary) until they exceed configurable thresholds
- Large collections use hash tables and skip lists for O(1) and O(log N) operations respectively
- LRU/LFU share the same 24 bits, selected by the configured eviction policy
- Shared integer pool (0–9999) eliminates allocations for the most common numeric values
Chapter 5 examines SDS — the Simple Dynamic String — which is the byte-level foundation that robj's string encodings are built on.