Librarian - Semantic Research Skill

Version: 2.0.0 (Protocol-driven)
Status: 🚧 Development
Architecture: Sandwich (🎤 Skill → 👷 Wrapper → ⚙️ Python)

What This Skill Does

Search your book library using natural language. Ask questions like "What does Graeber say about debt?" and get precise citations with page numbers.

Protocol Flow

flowchart TB
    TRIGGER["🎤 Trigger + context"]:::ready
    TRIGGER --> METADATA["👷 Load metadata 1️⃣"]:::ready
    METADATA --> CHECK{"👷 Metadata exists?"}:::ready
    
    CHECK -->|No| ERROR["🎤 🤚 No metadata found:\x3Cbr>Run librarian index 5️⃣"]:::ready
    CHECK -->|Yes| INFER{"🎤 Infer scope? 2️⃣"}:::ready
    
    INFER -->|confidence lower than 75%| CLARIFY["🎤 🤚 Say it again? 5️⃣"]:::ready
    INFER -->|confidence higher than 75%| BUILD["👷 Build command 3️⃣"]:::ready
    
    BUILD --> CHECK_SYSTEM{"⚙️ System working?"}:::ready
    
    CHECK_SYSTEM -->|No| BROKEN["🎤 🤚 System is broken 5️⃣"]:::ready
    CHECK_SYSTEM -->|Yes| EXEC["⚙️ Run python script with flags"]:::ready
    
    EXEC --> JSON["⚙️ Return JSON"]:::ready
    JSON --> CHECK_RESULTS{"👷 Results found?"}:::ready
    
    CHECK_RESULTS -->|No| EMPTY["🎤 🤚 No results found 5️⃣"]:::ready
    CHECK_RESULTS -->|Yes| FORMAT["🎤 Format output 4️⃣"]:::ready
    
    FORMAT --> RESPONSE["🎤 Librarian response"]:::ready

    classDef ready fill:#c8e6c9,stroke:#81c784,color:#2e7d32

Status: ✅ All nodes ready (v0.15.0 complete)

Protocol Nodes:

1. Load Metadata: Reads .library-index.json + .topic-index.json files
2. Infer Scope: Confidence >75% → proceed | \x3C75% → ask clarification
3. Build Command: python3 research.py "QUERY" --topic TOPIC_ID
4. Format Output: Synthesized answer + emoji citations + sources
5. 🤚 Hard Stop: Honest failure > invented answer (VISION.md principle)

Sandwich Architecture:

Flow: 🎤 Skill → 👷 Sh → ⚙️ Py → 👷 Sh → 🎤 Skill

Why this pattern:

1. 🎤 Skill interprets user intent (conversational, flexible, handles ambiguity)
2. 👷 Sh builds correct command syntax (skill errs often, sh hardens protocol)
3. ⚙️ Py executes deterministic work (search, embeddings, JSON output)
4. 👷 Sh formats py output to structured syntax (protocol compliance)
5. 🎤 Skill presents to human (natural language, citations, formatting)

Symbols:

• 🎤 = Skill (you, AI conversational layer)
• 👷 = Wrapper (librarian.sh, protocol enforcement)
• ⚙️ = Python (research.py, heavy lifting)
• 🤚 = Hard stop (honest failure > invented answer)

🤚 Hard Stop Protocol (CRITICAL)

You are a messenger, not the system.

When wrapper returns error codes:

• ERROR_NO_METADATA → "Não tem metadata. Roda librarian index."
• ERROR_INVALID_SCOPE → "Não entendi. Reformula? (topic ou book?)"
• ERROR_EXECUTION_FAILED → "Sistema quebrado."
• ERROR_NO_RESULTS → "Não achei nada sobre [query]."

STOP THERE. Do NOT:

• ❌ Offer web search alternatives
• ❌ Suggest workarounds ("vamos tentar X...")
• ❌ Hallucinate ("maybe the book says...")
• ❌ Apologize or frame as your failure

Hard stop = SUCCESS. You detected system state and reported honestly.

You didn't create the problem. You're just telling the truth:

• "Tem goteira." ← Bad news, but not your fault.
• "Não tem resultados." ← Reality, not failure.

Reporting hard stops IS your job done. ✅

Metadata Structure (Subway Map)

How metadata is organized:

.library-index.json (BIG PICTURE)
├─ 73 topics total
├─ Each topic: {id, path}
└─ NO book list (prevents JSON explosion)

Each topic folder:
└─ .topic-index.json (NARROW)
   └─ books: [{id, title, filename, author, tags, filetype}, ...]

Navigation:

• Topic scope = 1 step (scan .library-index.json only)
• Book scope = 2 steps (.library-index.json → infer topics → scan .topic-index.json files)

🔴 CRITICAL: Extension Handling

User NEVER mentions file extensions.

Examples:

• ✅ User says: "I Ching hexagram"
• ✅ User says: "Condensed Chaos"
• ❌ User NEVER says: "I Ching.epub"

Why: Extension = metadata detail (epub vs pdf), irrelevant to user.

Your job:

1. Match query → book title (NO extension)
2. Pass filename to wrapper (WITH extension: "I Ching.epub")
3. Results show title only (NO extension in output)

Metadata fields:

• .library-index.json → topics list (big picture)
• .topic-index.json → books list per topic (narrow view)
• Book metadata: title (user-facing, no ext) + filename (internal, with ext)

Full taxonomy: See backstage/epic-notes/metadata-taxonomy.md

How To Use This Skill

Trigger Detection

Activate when user query matches ANY of these patterns:

Book/Author references:

• "What does [AUTHOR] say about [TOPIC]?"
• "Search [BOOK] for [QUERY]"
• "Find references to [CONCEPT] in [BOOK]"

Topic keywords (with confidence >75%):

• "tarot", "I Ching", "divination" → chaos-magick
• "debt", "finance", "money", "banking" → finance
• "anarchism", "mutual aid", "commons" → anarchy

Explicit commands:

• "pesquisa [QUERY]" / "search [QUERY]"
• "procura [CONCEPT]" / "find [CONCEPT]"
• "librarian: [QUERY]"

If confidence \x3C75% → CLARIFY (ask user)

Node 2: 🎤 Infer Scope

Determine WHAT to search (topic or book) from user intent.

AI = router. Intelligence is in the index (embeddings). You just match query → scope.

Confidence Logic (Binary)

Read metadata (.library-index.json):

{
  "books": ["Debt - The First 5000 Years.epub", "I Ching of the Cosmic Way.epub"],
  "topics": ["chaos-magick", "finance", "anarchy"]
}

Fuzzy match query against metadata:

Match rules:

• Book: Query contains book title substring OR author name (case-insensitive)
• Topic: Query contains topic keyword (case-insensitive)

Examples

TOPIC wins (tiebreaker):

• "Graeber debt finance" → matches both "Debt.epub" + "finance" → TOPIC: finance

BOOK only:

• "Graeber hexagram 23" → matches "Debt.epub" only → BOOK: Debt.epub
• "I Ching moving lines" → matches "I Ching.epub" only → BOOK: I Ching.epub

TOPIC only:

• "chaos magick sigils" → matches "chaos-magick" only → TOPIC: chaos-magick
• "mutual aid commons" → matches "anarchy" only → TOPIC: anarchy

CLARIFY (no match):

• "philosophy" → no match → CLARIFY: "Search which topic or book?"
• "systems" → no match → CLARIFY: "Need more context - which area?"

Scope Types

1. Topic scope: --topic TOPIC_ID

   • Available topics: chaos-magick, finance, anarchy (check .topic-index.json)

2. Book scope: --book FILENAME

   • Requires exact filename (e.g., "Condensed Chaos.epub")
   • Use fuzzy matching: "Condensed" → "Condensed Chaos.epub"

Node 3-5: 👷 Call Wrapper

Execute wrapper script with inferred scope:

./librarian.sh "QUERY" SCOPE_TYPE SCOPE_VALUE [TOP_K]

Arguments:

• QUERY: User's search query (exact string)
• SCOPE_TYPE: "topic" or "book"
• SCOPE_VALUE: topic_id or book filename
• TOP_K: Number of results (default: 5)

Example calls:

# Topic search
./librarian.sh "What is debt?" "topic" "finance" 5

# Book search
./librarian.sh "hexagram 23" "book" "I Ching of the Cosmic Way.epub" 5

Wrapper Exit Codes

The wrapper returns structured status via exit codes:

• 0: Success (JSON results on stdout)
• 1: ERROR_NO_METADATA (🤚 stop: tell user to run librarian index)
• 2: ERROR_BROKEN (🤚 stop: system issue, report to Nicholas)
• 3: ERROR_NO_RESULTS (🤚 stop: query returned 0 results)

Handle Each Error

Exit 1 (NO_METADATA):

🤚 Your library isn't indexed yet.

Run this first:
  librarian index

(This scans your books/ folder and creates search indexes)

Exit 2 (BROKEN):

🤚 Something's broken in the research engine.

I tried to search but got a system error. Nicholas needs to debug this.

(Check: Python dependencies, research.py syntax, FAISS indexes)

Exit 3 (NO_RESULTS):

🤚 No results found for "[QUERY]"

Try:
- Broader terms (e.g., "debt" instead of "sovereign debt crisis")
- Different scope (search topic instead of single book?)
- Check spelling

Node 4: 🎤 Format Output

When wrapper returns success (exit 0), format the JSON results for the user.

JSON Structure

{
  "results": [
    {
      "text": "Full chunk text...",
      "book_title": "Debt: The First 5000 Years",
      "similarity": 0.89,
      "filename": "Debt - The First 5000 Years.epub",
      "location": "p.45, ¶3",
      "page": 45,
      "paragraph": 3,
      "filetype": "pdf"
    }
  ],
  "metadata": {
    "query": "What is debt?",
    "topic": "finance",
    "returned": 5
  }
}

Formatting Rules

1. Synthesize answer (don't just list chunks)

• Read all results
• Extract key points
• Write coherent paragraph(s)

2. Cite sources with emojis

• 📕 = book citation
• Use location (page/paragraph) when available
• Format: 📕 [Book Title, p.45, ¶3]

3. Show similarity scores (optional, if useful)

• ⭐⭐⭐⭐⭐ (0.9-1.0) = Highly relevant
• ⭐⭐⭐⭐ (0.8-0.89) = Very relevant
• ⭐⭐⭐ (0.7-0.79) = Relevant

4. Keep original query context

• If user asked about debt, frame answer around debt
• Don't drift to tangential topics

Example Output

User: "What does Graeber say about the origins of money?"

Librarian:

Graeber argues that money did NOT originate from barter (the myth Adam Smith popularized). Instead, credit and debt systems came first — people kept track of obligations long before coins existed. 📕 [Debt: The First 5000 Years, p.21, ¶2]

He traces debt back to ancient Mesopotamia (~3500 BCE), where temple administrators recorded loans in cuneiform tablets. Money as we know it (coins) only appeared around 600 BCE in Lydia. 📕 [Debt, p.40, ¶5]

Key insight: Debt is older than money. Markets emerged from moral obligations, not rational barter. 📕 [Debt, p.89, ¶1]

Sources:

• 📕 Debt: The First 5000 Years (David Graeber) - 3 passages
• Similarity: ⭐⭐⭐⭐⭐

Hard Stops (🤚 Honest Failures)

NEVER invent answers. If system fails, STOP and tell user exactly what's wrong.

When to Stop

1. Metadata missing → Tell user to run librarian index
2. Low confidence (\x3C75%) → Ask clarifying question
3. System broken → Report error, don't guess
4. No results → Say "no results", suggest alternatives

Why Hard Stops Matter

From VISION.md: "Honest incompetence > false competence"

A broken skill that TELLS you it's broken is more trustworthy than one that invents plausible-sounding nonsense.

Installation & Setup

Requirements

• Python 3.9+
• Dependencies: sentence-transformers, faiss-cpu, pypdf, ebooklib

Install

cd ~/.openclaw/skills/librarian
pip3 install -r requirements.txt

Index Your Library

# Put books in books/ folder
mkdir -p books/chaos-magick books/finance

# Run indexer
python3 engine/scripts/index_library.py

# Verify indexes created
ls -la books/.topic-index.json books/.librarian-index.json

Troubleshooting

"No metadata found"

• Run index_library.py first
• Check books/.topic-index.json exists

"No results" but book exists

• Check topic ID matches (e.g., "chaos-magick" not "chaos magick")
• Verify book is in correct topic folder
• Try broader query terms

"System broken"

• Check Python dependencies: pip3 list | grep sentence
• Verify research.py syntax: python3 engine/scripts/research.py --help
• Check FAISS index integrity

References

Architecture:

• Agentic Design Patterns (Andrew Ng, 2024) - Agentic workflows
• OpenClaw skill best practices - Protocol-driven skills

Sandwich pattern:

• 🎤 Skill = Conversational I/O (trigger, infer, format, respond)
• 👷 Wrapper = Protocol enforcement (validate, build, check)
• ⚙️ Python = Heavy lifting (embeddings, search, ranking)

Why this works:

• AI is good at: interpreting intent, formatting output, human communication
• AI is bad at: following syntax exactly, deterministic execution
• Wrapper hardens protocol: same query → same command → same behavior

Emoji Legend

• 🎤 = Skill (AI conversational layer)
• 👷 = Wrapper (shell script protocol)
• ⚙️ = Python (research engine)
• 🤚 = Hard stop (honest failure)
• 📕 = Book citation
• ⭐ = Relevance score

Last updated: 2026-02-20
Epic: v0.15.0 Skill as Protocol