← Back to Skills Marketplace
lookupmark

Local RAG

by LookUpMark · GitHub ↗ · v1.9.1 · MIT-0
cross-platform ✓ Security Clean
208
Downloads
0
Stars
0
Active Installs
14
Versions
Install in OpenClaw
/install lookupmark-local-rag
Description
Semantic search over local files using all-MiniLM-L6-v2 embeddings and ms-marco-MiniLM-L-6-v2 cross-encoder reranking with ChromaDB and parent-child chunking...
README (SKILL.md)

Local RAG

Semantic search over indexed local files with parent-child chunking for precise retrieval with full context.

Architecture

Component Model Size
Embeddings sentence-transformers/all-MiniLM-L6-v2 ~80MB
Reranker cross-encoder/ms-marco-MiniLM-L-6-v2 ~80MB
Vector DB ChromaDB (persistent, cosine similarity, HNSW) varies
Chunking Parent-child —

Memory strategy: Embedding model loaded first → freed with gc.collect() → reranker loaded → freed after scoring. This keeps peak RAM ~400MB on ARM.

Chunking Strategy

  • Child chunks: 128 words, 24 overlap → embedded for semantic search
  • Parent chunks: 768 words → stored as full context, returned to user
  • When a child matches → its parent is returned, giving surrounding context

Running

All scripts must use the venv Python:

VENV=~/.local/share/local-rag/venv/bin/python

Indexing

# Incremental index (default — skips unchanged files via SHA-256 hash)
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index.py

# Re-index from scratch
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index.py --reindex

# Custom paths
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index.py --paths ~/Documenti ~/Progetti

# Batch indexing (per-subfolder with git checkpoints, for low-RAM systems)
bash ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/index-batch.sh

Querying

# Basic query
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/query.py "what are the termination clauses?"

# More results
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/query.py "Falcon LLM" --top-k 30 --top-n 5

# JSON output for programmatic use
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/query.py "transformer architecture" --json

# With timeout
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/query.py "deep learning" --timeout 60

Options:

  • --top-k N — Child candidates from vector search (default: 20)
  • --top-n N — Final parent results after reranking (default: 3)
  • --json — JSON output
  • --timeout N — Max seconds per query (default: 120)

Monitoring

$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py              # Status
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py --watch      # Auto-refresh
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py --log        # Logs
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py --errors     # Errors only
$VENV ~/.openclaw/workspace/skills/lookupmark-local-rag/scripts/monitor.py --git        # Git checkpoints

Supported Formats

Documents only (no code files):

  • Text: .txt, .md, .csv, .json, .yaml, .yml, .toml, .tex, .bib
  • Documents: .pdf (pdfminer.six), .docx (python-docx), .pptx

Excluded: .py, .js, .sh, .ipynb, .html, .css and all code files.

Limits (for 4GB ARM)

  • PDF max size: 5MB (larger PDFs cause OOM with pdfminer)
  • Max file size: 30MB
  • Embedding batch size: 1 (conservative)
  • Excluded dirs: .git, .venv, node_modules, __pycache__, labs, exercises, src, scripts, ablation, test*, fixtures

Storage

Path Purpose
~/.local/share/local-rag/chromadb/ ChromaDB data (git repo for rollback)
~/.local/share/local-rag/venv/ Python venv with dependencies
~/.local/share/local-rag/index.lock Prevents concurrent indexing
~/.local/share/local-rag/index-batch.log Batch indexing log
~/.local/share/local-rag/queries.log Query history log

Security

  • ALLOWED_ROOTS: Only ~/Documenti/github/thesis, ~/Documenti/github/polito, ~/Documenti, ~/Scaricati
  • BLOCKED_PATTERNS: .ssh, .gnupg, .env, credentials, tokens, .config/openclaw
  • Credentials directory is blacklisted — never indexed

Workflow

  1. Run index.py — builds/rebuilds the index (incremental via SHA-256 hash check)
  2. Run periodically to pick up new/changed files (daily cron recommended)
  3. Use query.py to search with natural language
  4. Results include: file path, relevance score, matched snippet, full parent context
  5. Check monitor.py for stats and queries.log for query history
Usage Guidance
This appears to be a legitimate local-document RAG tool. Before installing or running it: 1) ensure you have or will create the indicated Python venv and install dependencies (sentence-transformers, chromadb, pdfminer, python-docx, etc.); 2) be aware the first run will download ML models from Hugging Face (internet access); 3) the tool creates a ChromaDB and logs under ~/.local/share/local-rag — check and if desired change that path or back it up; 4) the skill appends queries (truncated) to ~/.local/share/local-rag/queries.log, so avoid searching highly sensitive secrets or adjust/disable logging; 5) verify ALLOWED_ROOTS and BLOCKED_PATTERNS in scripts/index.py match what you want indexed (the default only indexes ~/Documenti and ~/Scaricati and subpaths listed); and 6) note small documentation mismatches (supported extensions list vs code) and the skill provides no automated installer — review and run the scripts manually the first time to confirm behavior.
Capability Analysis
Type: OpenClaw Skill Name: lookupmark-local-rag Version: 1.9.1 The 'lookupmark-local-rag' skill is a legitimate implementation of a local Retrieval-Augmented Generation (RAG) system optimized for low-RAM ARM devices. It includes robust security controls in `scripts/index.py`, such as directory allow-listing (`ALLOWED_ROOTS`) and sensitive file blacklisting (`BLOCKED_PATTERNS` for `.ssh`, `.env`, etc.) to prevent accidental indexing of credentials. The use of Git in `scripts/index-batch.sh` for database checkpoints is a functional recovery mechanism for OOM errors, and no evidence of data exfiltration, unauthorized network access, or malicious prompt injection was found.
Capability Tags
crypto
Capability Assessment
✓ Purpose & Capability
Name/description (Local RAG for semantic search) match the included scripts (index.py, query.py, monitor.py, batch scripts). The code only operates on local files and a local ChromaDB, and uses expected components (sentence-transformers, cross-encoder, ChromaDB). No unrelated credentials, external service tokens, or cloud SDKs are requested.
ℹ Instruction Scope
SKILL.md and scripts limit indexing to specified home directories and explicitly blacklist typical credential dirs. The code reads system state (/proc/meminfo) and runs local commands (ps, du, git) for monitoring — appropriate for monitoring/indexing. Two minor scope mismatches: SKILL.md's 'Supported Formats' lists some extensions (e.g., .json, .yaml, .pptx) that are not all reflected in index.py's TEXT_EXTENSIONS/ALL_EXTENSIONS, and the SKILL.md expects a venv at ~/.local/share/local-rag/venv but the skill does not include an installer to create it. Also, queries are appended to a local queries.log (question truncated to 80 chars) — this is local logging and a privacy consideration but coherent with the tool's purpose.
✓ Install Mechanism
No install spec is provided (instruction-only), minimizing install-time risk. The code will download models from Hugging Face when first run (network activity) and expects a Python venv with dependencies; those are normal for this functionality and there are no suspicious external URLs or archive downloads in the files.
ℹ Credentials
The skill requests no environment variables or external credentials. It does require read access to user document directories and write access to ~/.local/share/local-rag for DB, venv, and logs — appropriate for its purpose. Consider that it will download ML models (internet access) and maintains a local queries.log (may contain user queries, albeit truncated).
✓ Persistence & Privilege
The skill does not request permanent platform-level privileges (always=false). It writes/creates files only under ~/.local/share/local-rag and does not modify other skills or system agent config. The included batch scripts use git for checkpointing inside the DB dir (local) which is reasonable for rollback behavior.
How to Use
  1. Make sure OpenClaw is installed (local or Docker)
  2. Run the install command in chat: /install lookupmark-local-rag
  3. After installation, invoke the skill by name or use /lookupmark-local-rag
  4. Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.9.1
Two-phase batch indexing, 25MB PDF limit, text-only/pdf-only modes, orphan detection fix
v1.9.0
Two-phase indexing: text first (25/batch), then PDFs (1/batch, up to 25MB). --text-only/--pdf-only flags.
v1.8.2
Batch indexing (--max-files), memory guards, GC, orphan detection fix, batch wrapper script
v1.8.1
Detailed indexing log: track added/updated/removed files with totals
v1.8.0
Memory guard, cached reranker model
v1.7.1
Fixed: args.json passed to timeout parameter (query.py main bug). Removed .html/.css from TEXT_EXTENSIONS to match SKILL.md documentation.
v1.7.0
Security: removed trust_remote_code=True. Fixed docstring (BGE-M3 → all-MiniLM). Narrowed pkill in batch script. Truncated query log for privacy.
v1.6.0
Security: removed auto pip install. Efficiency: dynamic batch size, embedding model caching, input sanitization.
v1.5.0
SKILL.md updated (correct models, limits, storage map). query.py: query logging, timeout flag. index.py: silenced FontBBox warnings.
v1.4.0
Added monitoring script (progress, logs, errors, git checkpoints, system stats)
v1.3.0
Security: path whitelist, blocked patterns, disk check. Exclude code files, max 30MB. pdfminer.six instead of pdfplumber.
v1.2.0
Fixed ChromaDB v1 compat ( filter), disabled reranker on ARM, batch size 1 for low RAM
v1.1.0
Parent-child chunking, GPU support, file lock, skip code files, Italian paths
v1.0.0
Initial release of local-rag: semantic search for local files. - Enables semantic search across local PDFs, DOCX, TXT, MD, and common code/text files. - Uses BGE-M3 embeddings and BGE-RERANKER-LARGE reranker for accurate retrieval. - Employs parent-child chunking: searches small chunks, returns larger context chunks. - Persistent local vector database (ChromaDB); CLI tools provided for indexing and searching. - Supports natural language queries and custom file path indexing.
Metadata
Slug lookupmark-local-rag
Version 1.9.1
License MIT-0
All-time Installs 0
Active Installs 0
Total Versions 14
Frequently Asked Questions

What is Local RAG?

Semantic search over local files using all-MiniLM-L6-v2 embeddings and ms-marco-MiniLM-L-6-v2 cross-encoder reranking with ChromaDB and parent-child chunking... It is an AI Agent Skill for Claude Code / OpenClaw, with 208 downloads so far.

How do I install Local RAG?

Run "/install lookupmark-local-rag" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.

Is Local RAG free?

Yes, Local RAG is completely free, licensed under MIT-0. You can download, install and use it at no cost.

Which platforms does Local RAG support?

Local RAG is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).

Who created Local RAG?

It is built and maintained by LookUpMark (@lookupmark); the current version is v1.9.1.

More Skills
self-improving agent
Captures learnings, errors, and corrections to enable continuous improvement. Use when: (1) A command or operation fails unexpectedly, (2) User corrects Clau...
⬇ 463368 · by pskoett
Skill Vetter
Security-first skill vetting for AI agents. Use before installing any skill from ClawdHub, GitHub, or other sources. Checks for red flags, permission scope, and suspicious patterns.
⬇ 259467 · by spclaudehome
Polymarket
Query Polymarket prediction markets. Check odds, find trending markets, search events, track price movements.
⬇ 232503 · by joelchance
Self-Improving + Proactive Agent
Self-reflection + Self-criticism + Self-learning + Self-organizing memory. Agent evaluates its own work, catches mistakes, and improves permanently. Use when...
⬇ 200457 · by ivangdavila
Github
Interact with GitHub using the `gh` CLI. Use `gh issue`, `gh pr`, `gh run`, and `gh api` for issues, PRs, CI runs, and advanced queries.
⬇ 191163 · by steipete
ontology
Typed knowledge graph for structured agent memory and composable skills. Use when creating/querying entities (Person, Project, Task, Event, Document), linkin...
⬇ 190189 · by oswalpalash

💬 Comments