Slice Internals: Growth Strategy and Memory Traps

Chapter 19: Slice Internals: Growth Strategy and Memory Traps

If asked to name the most "deceptively simple, yet genuinely deep" data structure in Go, the slice earns that title without contest. Every Go programmer has written append(s, x), yet few can accurately answer these questions:

• After append, does the original slice variable s still point to the same memory?
• Why can two slices share the same backing array, so modifying one affects the other?
• Go 1.18 changed append's growth algorithm — what was the old algorithm, what is the new one, and why was the change made?
• How do you implement zero-copy conversion between string and []byte using unsafe?

This chapter begins with the three-field SliceHeader, dives into the growslice growth logic, dissects the memory traps that catch even experienced Go developers, and finally explores the boundaries of unsafe black magic and reflection-based manipulation.

Level 1: Fundamentals — The Essential Distinction Between Array and Slice

Why Slices Exist

Before understanding slices, you must understand the limitations of Go arrays.

Arrays are value types, and size is part of the type:

a := [5]int{1, 2, 3, 4, 5}
b := a  // b is a complete copy of a; modifying b does not affect a

// Array size is part of the type: [5]int and [6]int are different types
var fn func([5]int)
// fn(a) ← valid
// fn([6]int{}) ← compile error: type mismatch

This means: you cannot write a function accepting "an integer array of any size." Each distinct array size is a different type. Passing a large array to a function also triggers a full value copy, which can be expensive.

Slices solve all of arrays' limitations:

• Variable size: append can grow a slice
• Type does not encode size: []int can reference an integer sequence of any length
• Lightweight to pass: passing a slice copies only 24 bytes of the header (on 64-bit systems), not the underlying data

The essence of a slice is a view into an underlying array, not a container of data itself. Understanding this "view" concept is the key to avoiding slice traps.

The Relationship Between Arrays and Slices

Backing array:
  ┌───┬───┬───┬───┬───┬───┬───┬───┐
  │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │ 8 │
  └───┴───┴───┴───┴───┴───┴───┴───┘
  [0] [1] [2] [3] [4] [5] [6] [7]

s1 := arr[1:5]  // ptr→arr[1], len=4, cap=7
  ┌───────────────────┐
  │ ptr → arr[1]      │
  │ len = 4           │
  │ cap = 7           │
  └───────────────────┘

s2 := arr[3:6]  // ptr→arr[3], len=3, cap=5
  ┌───────────────────┐
  │ ptr → arr[3]      │
  │ len = 3           │
  │ cap = 5           │
  └───────────────────┘

s1 and s2 share the same backing array:
modifying s1[2] (i.e., arr[3]) also changes s2[0]!

Level 2: Internals — SliceHeader and growslice

SliceHeader: Three Fields

In Go's memory, a slice variable occupies 24 bytes (on a 64-bit system), composed of three fields:

type SliceHeader struct {
    Data uintptr  // pointer to the backing array (8 bytes)
    Len  int      // current number of elements (8 bytes)
    Cap  int      // total capacity from the Data pointer to the end (8 bytes)
}

The invariant: 0 <= Len <= Cap. Cap is the number of elements from the Data pointer to the end of the backing array. Len is the number of "visible" elements.

Verifying the three fields:

s := make([]int, 3, 5)
// len(s)=3, cap(s)=5
// s[0]=0, s[1]=0, s[2]=0
// s[3] and s[4] exist in the backing array but are invisible through s

// Use reflect.SliceHeader to inspect internal fields
sh := (*reflect.SliceHeader)(unsafe.Pointer(&s))
fmt.Printf("Data: %x, Len: %d, Cap: %d\n", sh.Data, sh.Len, sh.Cap)

append and Growth: Three Scenarios

The behavior of append(s, elems...) depends on the slice's remaining capacity (cap - len):

Scenario 1: Sufficient cap, no growth needed

s := make([]int, 2, 5)  // len=2, cap=5
s2 := append(s, 99)     // len=3, cap=5
// s2 points to the same backing array!
// s2[2] == 99; though invisible through s (len=2), the memory at index 2 was written

fmt.Println(s[:3])  // [0 0 99]  ← visible via re-slicing

Scenario 2: Insufficient cap, growth triggered

s := []int{1, 2, 3}  // len=3, cap=3
s2 := append(s, 4)   // growth triggered!
// s2 points to a newly allocated backing array
// s still points to the old backing array
// Modifying s2 no longer affects s, and vice versa

Scenario 3: Bulk append

a := []int{1, 2}
b := []int{3, 4, 5}
c := append(a, b...)  // expand all elements of b and append to a

growslice: The Algorithm Before and After Go 1.18

growslice is the runtime function responsible for slice growth (runtime/slice.go). In Go 1.17 and earlier, the growth rule was:

Old algorithm (Go 1.17 and below):
  if newcap > 2 * oldcap:
      newcap = newcap      // use the demanded capacity directly
  else if oldcap < 1024:
      newcap = 2 * oldcap  // double
  else:
      for newcap < cap:
          newcap += newcap / 4  // grow by 25% each iteration

This algorithm has a noticeable discontinuity: doubling applies up to cap=1023, then switches to 25% growth at cap=1024, producing a sudden kink in the growth curve.

Go 1.18 introduced a smoother growth strategy:

New algorithm (Go 1.18+):
  1. If newcap > 2 * oldcap: newcap = newcap
  2. Otherwise:
     threshold = 256
     if oldcap < threshold:
         newcap = 2 * oldcap
     else:
         while newcap < cap:
             // smooth transition from 2x toward 1.25x
             newcap += (newcap + 3 * threshold) / 4

The key improvement: doubling for small capacities, smoothly curving toward ~1.25x growth for large capacities, eliminating the sharp transition at 1024.

Comparing the growth sequences (Go 1.17 vs Go 1.18, when one more element than the current capacity is needed):

oldcap → newcap

Go 1.17:
1→2→4→8→16→32→64→128→256→512→1024→1280→1600→2000...
                                    ↑
                            abrupt jump from 2x to 1.25x

Go 1.18:
1→2→4→8→16→32→64→128→256→512→848→1280→1792→2304...
                                  ↑
                            smooth transition

Why smooth growth matters: overly aggressive growth (always 2x) wastes memory; overly conservative growth (always 1.25x) triggers frequent reallocations (each append may trigger one). A smooth curve balances both concerns.

Memory Layout After Growth

Before growth (cap=4, len=4):
  Data → ┌───┬───┬───┬───┐
         │ 1 │ 2 │ 3 │ 4 │
         └───┴───┴───┴───┘

s2 := append(s, 5)  // needs cap=5, triggers growth (new cap ≈ 8)

After growth (old array abandoned):
  old Data → ┌───┬───┬───┬───┐   (awaiting GC)
              │ 1 │ 2 │ 3 │ 4 │
              └───┴───┴───┴───┘

  new Data → ┌───┬───┬───┬───┬───┬───┬───┬───┐
              │ 1 │ 2 │ 3 │ 4 │ 5 │   │   │   │
              └───┴───┴───┴───┴───┴───┴───┴───┘
  len=5, cap=8

s still points to old Data (len=4, cap=4)
s2 points to new Data (len=5, cap=8)

This is the most important insight about slices: append may return a new slice sharing the backing array with the original, or it may return a slice with an entirely new backing array. It depends on whether growth was triggered. Always assign the return value of append back to the slice variable: s = append(s, x), never use s after a bare append(s, x).

copy Semantics

copy(dst, src) copies elements from src into dst, returning the actual count copied (min(len(dst), len(src))). It allocates no new memory, does not change len or cap, and handles overlapping backing arrays correctly (behaving like C's memmove).

src := []int{1, 2, 3, 4, 5}
dst := make([]int, 3)
n := copy(dst, src)
fmt.Println(n, dst)  // 3, [1 2 3]

// copy can also shift elements within the same slice
s := []int{1, 2, 3, 4, 5}
copy(s[1:], s[2:])  // shift s[2:] left by one position
fmt.Println(s)      // [1 3 4 5 5] (last 5 is stale data)

Level 3: Code Patterns — Common Traps and Correct Usage

Trap 1: append Causing Unexpected Shared Mutations

This is the most common slice trap in Go and the source of some of the hardest-to-debug bugs:

// Scenario: a function receives a slice, appends an element, returns it
func addElement(s []int) []int {
    return append(s, 99)
}

base := make([]int, 3, 5)  // len=3, cap=5 (has room!)
base[0], base[1], base[2] = 1, 2, 3

result := addElement(base)
// result is a view into the same backing array as base with a larger len
// result[3] == 99

result2 := addElement(base)
// base still has capacity! result2 also uses the same backing array
// result2[3] == 99, written to the same position as result[3]

// If we then append to result2:
result3 := append(result2, 100)
// result3[4] = 100, written to backing array position [4]
// result and result2 may see this "ghost" element via re-slicing

Solution: use the full slice expression or explicit copy when you need an independent copy:

// Method 1: three-index slice to cap the capacity, forcing growth on next append
func addElementSafe(s []int) []int {
    // s[:len(s):len(s)] caps capacity at len, so the next append must allocate new memory
    return append(s[:len(s):len(s)], 99)
}

// Method 2: explicit copy to create an independent copy
func addElementCopy(s []int) []int {
    clone := make([]int, len(s), len(s)+1)
    copy(clone, s)
    return append(clone, 99)
}

base := make([]int, 3, 5)
base[0], base[1], base[2] = 1, 2, 3

r1 := addElementSafe(base)
r2 := addElementSafe(base)
// r1 and r2 now have independent backing arrays
fmt.Println(r1, r2)  // [1 2 3 99] [1 2 3 99] — no interference

Trap 2: Slice of Struct vs Slice of Pointer

type User struct {
    Name string
    Age  int
}

// Approach 1: []User (slice of struct)
users := []User{{"Alice", 30}, {"Bob", 25}}
for _, u := range users {
    u.Age += 1  // u is a copy! Modification has no effect
}
fmt.Println(users[0].Age)  // still 30

// Fix: use index
for i := range users {
    users[i].Age += 1  // modifies the element in the slice directly
}
fmt.Println(users[0].Age)  // 31

// Approach 2: []*User (slice of pointer)
pUsers := []*User{{"Alice", 30}, {"Bob", 25}}
for _, u := range pUsers {
    u.Age += 1  // u is a pointer; modification is effective
}
fmt.Println(pUsers[0].Age)  // 31

When to use []T vs []*T:

Three-Index Slice: a[low:high:max]

Go 1.2 introduced three-index slice syntax, allowing precise control over a new slice's capacity:

a := []int{1, 2, 3, 4, 5, 6, 7, 8}

// Two-index slice: cap inherited from original slice's remaining cap
s1 := a[2:5]      // Data=&a[2], len=3, cap=6 (to end of array)

// Three-index slice: explicit cap control
s2 := a[2:5:6]    // Data=&a[2], len=3, cap=4 (a[2] to a[5], exclusive of a[6])
// max=6 means the new slice's cap extends to a[5] (index 5, not included)

// Utility of capping capacity:
append(s2, 99)         // must allocate new array (cap=4, len=3, one slot free but gone)
append(s2, 99, 100)    // exceeds cap, must grow, new array allocated

The three-index slice is a powerful tool for library design. When your function returns a subset of an input slice, capping the capacity prevents callers' append calls from silently poisoning the memory beyond the returned slice:

func getSlice(data []byte) []byte {
    // Return only the first 10 bytes, cap-limited to prevent callers'
    // append from affecting data[10:]
    return data[:10:10]
}

copy vs append for Merging Slices

a := []int{1, 2, 3}
b := []int{4, 5, 6}

// Method 1: append (idiomatic, concise)
merged := append(a, b...)
// Warning: if a has enough cap, merged shares a's backing array!

// Safe version: copy a first, then append
merged = append(append([]int(nil), a...), b...)

// Method 2: make + copy (explicit, predictable)
merged2 := make([]int, len(a)+len(b))
copy(merged2, a)
copy(merged2[len(a):], b)

Which is faster? For large slices, the copy version is generally faster: it allocates exactly once (while append may allocate multiple times) and uses memmove for bulk copying, allowing the CPU to exploit SIMD instructions.

Performance benchmark (merging two slices of 10,000 elements each):

BenchmarkAppend-8    500000    3.2 µs/op
BenchmarkCopy-8      800000    1.8 µs/op
// copy is ~1.8x faster

Level 4: Advanced — Zero-Copy Conversion, Reflection and Memory Pinning

Zero-Copy Conversion Between string and []byte

Standard conversions between string and []byte in Go trigger memory copies:

b := []byte("hello")           // allocates new memory, copies bytes
s := string([]byte{104, 105})  // allocates new memory, copies bytes

In hot paths (such as HTTP handlers parsing request bodies), this copying is a performance bottleneck. You can use unsafe for zero-copy conversion:

import "unsafe"

// []byte to string, zero-copy (read-only; do not modify the source []byte)
func BytesToString(b []byte) string {
    return *(*string)(unsafe.Pointer(&b))
}

// string to []byte, zero-copy (dangerous! modifying the returned []byte
// violates Go's string immutability guarantee)
func StringToBytes(s string) []byte {
    sh := (*reflect.StringHeader)(unsafe.Pointer(&s))
    bh := reflect.SliceHeader{
        Data: sh.Data,
        Len:  sh.Len,
        Cap:  sh.Len,
    }
    return *(*[]byte)(unsafe.Pointer(&bh))
}

These functions exploit the memory layout similarity between string and SliceHeader:

string header:
  ┌──────────────────────┬──────┐
  │ Data *byte (8 bytes) │ Len  │
  └──────────────────────┴──────┘

SliceHeader:
  ┌──────────────────────┬──────┬──────┐
  │ Data *byte (8 bytes) │ Len  │ Cap  │
  └──────────────────────┴──────┴──────┘

BytesToString: reinterprets the first 16 bytes of a SliceHeader as a StringHeader
StringToBytes: appends Cap=Len to a StringHeader to construct a SliceHeader

Safety rules for zero-copy conversion:

1. BytesToString: if the original []byte is modified, the string's content changes (violating Go's string immutability, causing undefined behavior). Only use when the []byte will not be modified afterward.
2. StringToBytes: never modify the returned []byte — doing so would overwrite a read-only string literal segment, causing a segfault. Only use when you are certain no modifications will be made.

Go 1.20 introduced officially supported zero-copy APIs, eliminating the need for hand-written unsafe:

// Go 1.20+ (officially recommended, safe)
import "unsafe"

s := "hello"
b := unsafe.Slice(unsafe.StringData(s), len(s))  // string → []byte, zero-copy

b2 := []byte{104, 101, 108, 108, 111}
s2 := unsafe.String(&b2[0], len(b2))  // []byte → string, zero-copy

reflect.SliceHeader Manipulation

reflect.SliceHeader allows runtime inspection and manipulation of a slice's internal structure:

import (
    "fmt"
    "reflect"
    "unsafe"
)

func inspectSlice(s []int) {
    sh := (*reflect.SliceHeader)(unsafe.Pointer(&s))
    fmt.Printf("Data=0x%x, Len=%d, Cap=%d\n", sh.Data, sh.Len, sh.Cap)
}

// Wrapping an arbitrary memory address as a slice
func wrapMemory(ptr uintptr, length int) []byte {
    sh := reflect.SliceHeader{
        Data: ptr,
        Len:  length,
        Cap:  length,
    }
    return *(*[]byte)(unsafe.Pointer(&sh))
}

// Example: accessing memory returned by C (CGo scenario)
// cPtr := C.malloc(1024)
// s := wrapMemory(uintptr(unsafe.Pointer(cPtr)), 1024)

Note: starting with Go 1.17, reflect.SliceHeader and reflect.StringHeader are quasi-deprecated. The recommended replacements are the four built-ins introduced in Go 1.17/1.20: unsafe.Slice, unsafe.SliceData, unsafe.String, and unsafe.StringData.

growslice Assembly Deep Dive

Go's growslice is partially implemented in assembly on AMD64. Understanding its calling convention helps with performance analysis:

// Scenarios that trigger growslice
s := make([]int, 0, 4)
for i := 0; i < 100; i++ {
    s = append(s, i)  // growslice triggered at i=4,8,16,32,64
}

// Use go build -gcflags="-m" to inspect escape analysis and inlining decisions
// go tool compile -S main.go | grep growslice
// Shows the call sites for growslice

The main steps of each growslice call:

growslice(oldPtr, newLen, oldCap, num, et):
  1. Compute newCap (using the algorithm above)
  2. Check et.ptrdata == 0 (does element type contain pointers?)
     - No pointers: mallocgc, no GC scanning needed
     - Has pointers: mallocgc, write GC bitmap
  3. memmove(new_data, old_data, oldLen * elemsize)
  4. Return (new_ptr, newLen, newCap)

Step 2 is a subtle performance detail: growing a slice containing pointers is slower than growing one without pointers, because the GC must record the pointer locations in newly allocated memory. This is one reason why append to []int is faster than append to []*int.

Memory Pinning: The Large Backing Array Trap

A slice's backing array is not reclaimed by the GC as long as any slice (or pointer) references it. A common hidden cost:

// Read a 1 GB file
data, _ := os.ReadFile("large_file.dat")  // []byte, len=1GB

// We only need the first 100 bytes
header := data[:100]  // header's backing array is still that 1 GB!

// Even if data becomes unreachable, header's reference keeps the 1 GB alive
data = nil  // ineffective! backing array still referenced by header

The correct approach: explicit copy to sever the reference to the large backing array:

data, _ := os.ReadFile("large_file.dat")

// Create an independent copy; release reference to the large backing array
header := make([]byte, 100)
copy(header, data[:100])
data = nil  // now the 1 GB can be GC'd

// Or more concisely (append([]byte(nil), ...) always allocates new memory):
header = append([]byte(nil), data[:100]...)
data = nil

This problem is very common in real Go engineering, especially when processing large files, network packets, or database query results.

Trap: Functions Returning Slices

// Concerning: returning a slice of a local array
func questionableFunc() []int {
    arr := [3]int{1, 2, 3}  // local array
    return arr[:]            // returns slice pointing to local array
    // Go's escape analysis will allocate arr on the heap — technically safe
    // but the intent is unclear
}

// Preferred: explicit allocation makes intent clear
func clearFunc() []int {
    return []int{1, 2, 3}  // compiler allocates on heap directly, intent is obvious
}

Go's escape analysis does handle "local array referenced externally" by allocating it on the heap, so there are no dangling pointers. But explicit heap allocation is preferable for readability.

Performance Optimization: append Best Practices

Pre-allocating Capacity

When the final size of a slice is known (or has a reasonable upper bound), pre-allocate capacity:

// Bad: multiple growth events
var s []int
for i := 0; i < 10000; i++ {
    s = append(s, i)  // ~13 growslice calls (2,4,8,16,...,8192,16384)
}

// Good: single allocation
s := make([]int, 0, 10000)
for i := 0; i < 10000; i++ {
    s = append(s, i)  // zero growslice calls
}

// Performance difference:
// Bad:  ~280 µs/op, 13 GC pressure events
// Good: ~ 32 µs/op, 0 additional allocations

For inputs from channels or unknown sizes, at least provide a reasonable initial capacity estimate:

// Instead of:
var results []Result
for item := range ch {
    results = append(results, process(item))
}

// Prefer:
results := make([]Result, 0, expectedSize)
for item := range ch {
    results = append(results, process(item))
}

Using the slices Package (Go 1.21+)

Go 1.21's slices standard library provides generic slice operations:

import "slices"

s := []int{3, 1, 4, 1, 5, 9, 2, 6}

// Sort
slices.Sort(s)  // [1 1 2 3 4 5 6 9]

// Binary search
idx, found := slices.BinarySearch(s, 5)
fmt.Println(idx, found)  // 5, true

// Reverse
slices.Reverse(s)

// Containment check
fmt.Println(slices.Contains(s, 7))  // false

// Deduplication (requires sorted input)
slices.Sort(s)
s = slices.Compact(s)  // remove adjacent duplicates

The Compile-Time vs Runtime Trade-off

For compute-critical paths where even a single allocation matters, knowing whether append will trigger growslice or not is essential. Use go build -gcflags="-m" to inspect compiler decisions:

$ go build -gcflags="-m" ./...
# Relevant output lines:
./main.go:15:10: make([]int, 0, n) does not escape  ← allocated on stack
./main.go:20:14: append(s, i) does not escape       ← no heap allocation
./main.go:25:12: []int literal escapes to heap       ← heap allocation

Stack allocation is an order of magnitude faster than heap allocation and produces no GC pressure. The compiler's escape analysis is conservative — it prefers correctness over optimization — so there are cases where you know something is stack-safe but the compiler allocates it on the heap. In those extreme cases, sync.Pool can reduce allocation pressure:

var pool = sync.Pool{
    New: func() interface{} {
        b := make([]byte, 0, 4096)
        return &b
    },
}

func handler() {
    bp := pool.Get().(*[]byte)
    buf := (*bp)[:0]  // reset length, keep capacity

    buf = append(buf, "hello"...)
    // ... use buf ...

    *bp = buf
    pool.Put(bp)  // return to pool; next Get() reuses this allocation
}

Summary

Slices are Go's most important data structure, but beneath their "value type" surface lies "reference type" substance. Understanding SliceHeader's three fields (ptr, len, cap), understanding append's three scenarios (sufficient cap, insufficient cap, bulk append), and understanding growslice's smooth growth algorithm in Go 1.18+ are the foundations of writing correct Go code.

Two core principles:

1. Always use the return value of append: write s = append(s, x), never use a bare append(s, x) and continue using s as if it were updated.
2. Explicitly copy when you need an independent copy: never assume a slice operation does not share memory, especially across function boundaries.

Master these, and you have mastered the most critical corner of Go's memory model.