LoRA for Consistent Characters

Ch09 Consistent Characters: LoRA Training & IP Reuse

The biggest pain point in AI image generation is not image quality — it is character consistency. The same male lead looks different in every new scene, completely breaking audience immersion. This chapter explains why this happens at the model level, and provides two practical solutions: deep-customization LoRA training, and the zero-training IP-Adapter fast track.

Why AI Images Lack Character Consistency by Default

Diffusion models like Stable Diffusion and Flux generate images by learning statistical patterns from billions of images and then "guessing" the most likely visual output for a given text prompt. The critical issue: the model has no memory of the previous image. Every generation is an independent random process. Even identical prompts produce noticeably different outputs each time.

For landscape art this is fine, but for short dramas requiring the same character to appear across many different scenes, this is a fatal flaw. When audiences see the male lead looking different between episodes, immersion breaks instantly.

LoRA: How It Works

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique. Instead of retraining the full base model (which would require hundreds of GB of compute), LoRA adds small "adapter" modules to specific model layers that learn only the target character's visual features. The resulting LoRA file is typically just 50-300MB, but can dramatically reshape the model's output for that character.

[NOTE] Analogy: The base model is a skilled actor who can play any role. LoRA is the character bible you hand them — detailed notes on how the character looks, moves, and presents in any situation. With that reference in hand, the "actor" stays in character across every scene.

Training Data Preparation

The quality ceiling of your LoRA is determined by your training data. Richer, more diverse data produces a LoRA that generalizes better across scenes and poses.

Image Count & Quality Requirements

• Minimum viable: 15-20 images — basic appearance captured, limited generalization

• Recommended: 30-50 images — covers multiple angles, expressions, and lighting

• Ideal: 80-100 images — full generalization across poses and settings

Diversity Requirements

• Angles: Front, 45-degree left/right, profile, top-down, low-angle

• Expressions: Neutral, smile, serious, surprised, sad

• Lighting: Natural, warm indoor, dramatic side light, backlit, low-light

• Backgrounds: 30% simple/solid backgrounds, 70% varied scene backgrounds

• Outfits: At least 3-4 different costume styles

Liblib.ai Training Steps

[Liblib.ai — Recommended Parameters]

Base Model: Realistic Vision V5.1 VAE
Training Steps: 1500-2500 (num_images × 40-50)
Learning Rate: 0.0001 (portrait recommended)
Network Dim: 32 or 64
Network Alpha: 16 or 32
Batch Size: 2
Resolution: 512x512 or 768x768
Trigger Word: custom unique token e.g. "mychar001"

IP-Adapter: Zero-Training Character Consistency

If you need to reuse an existing character image without training a LoRA, IP-Adapter is the faster solution. It encodes a reference image as a vector and feeds it as an additional guidance signal into the generation process, making the model reference the visual features of your specified image.

[TIP] IP-Adapter best practice: Use a clear, near-frontal, high-resolution photo as your reference image for best results. The reference image's art style will bleed into the output, so use a photorealistic reference if you want photorealistic output.

Prompt Techniques for Cross-Scene Character Stability

Even with LoRA, character appearance can drift slightly when switching scenes. These prompt techniques further lock character features:

[Character Anchor Prompts]

-- Boost trigger word weight --
(mychar001:1.3), same character, consistent appearance,

-- Anchor key features --
identical face, same hairstyle, same facial features, consistent character design

-- Negative drift prevention --
different person, character change, inconsistent appearance

Building a Character Asset Library

[Folder Structure]

drama-assets/
├── characters/
│   ├── male_lead/
│   │   ├── lora/
│   │   │   └── male_lead_v1.safetensors
│   │   ├── reference/
│   │   ├── expressions/
│   │   │   ├── neutral.png
│   │   │   ├── angry.png
│   │   │   └── smile.png
│   │   └── prompts.txt
│   └── female_lead/
└── episodes/

[TIP] Chapter Action Checklist:

1. Collect 30 multi-angle, multi-expression images of your lead character;
2. Complete one LoRA training run on Liblib.ai;
3. Verify consistency by generating the same character in 5 different scenes;
4. Install ComfyUI + IP-Adapter and test with a single reference image;
5. Set up the drama-assets/ folder structure and archive your existing materials.

← PreviousCh08 AI Image Generation Next →Ch10 AI Video Generation