Description

Automation skill for Claw Compactor.

README (SKILL.md)

name: claw-compactor version: "1.0.0" description: > Claw Compactor - 6-layer token compression skill for OpenClaw agents. Cuts workspace token spend by 50–97% using deterministic rule-engines plus Engram: a real-time, LLM-driven Observational Memory system. Run at session start for automatic savings reporting.

triggers:

"compress memory", "compress workspace", "save tokens", "token savings", "compress context", "run engram", "engram observe", "engram reflect", "memory compression", "benchmark compression"

Claw Compactor - OpenClaw Skill Reference

Name: Claw Compactor
Author: gandli-2025

Overview

Claw Compactor reduces token usage across the full OpenClaw workspace using 6 compression layers:

1, Name=Rule Engine, Cost=Free, Notes=Dedup, strip filler, merge sections 2, Name=Dictionary Encoding, Cost=Free, Notes=Auto-codebook, $XX substitution 3, Name=Observation Compression, Cost=Free, Notes=Session JSONL → structured summaries 4, Name=RLE Patterns, Cost=Free, Notes=Path/IP/enum shorthand 5, Name=Compressed Context Protocol, Cost=Free, Notes=Format abbreviations 6, Name=Engram, Cost=LLM API, Notes=Real-time Observational Memory

Skill location: skills/claw-compactor/ Entry point: scripts/mem_compress.py Engram CLI: scripts/engram_cli.py

Auto Mode (Recommended - Run at Session Start)

python3 skills/claw-compactor/scripts/mem_compress.py \x3Cworkspace> auto

Automatically compresses all workspace files, tracks token counts between runs, and reports savings. Run this at the start of every session.

Core Commands

Full Pipeline (All Layers)

python3 scripts/mem_compress.py \x3Cworkspace> full Runs all 5 deterministic layers in optimal order. Typical: 50%+ combined savings.

Benchmark (Non-Destructive)

python3 scripts/mem_compress.py \x3Cworkspace> benchmark

JSON output:

python3 scripts/mem_compress.py \x3Cworkspace> benchmark --json Dry-run report showing potential savings without writing any files.

Layer 1: Rule-based compression

python3 scripts/mem_compress.py \x3Cworkspace> compress

Layer 2: Dictionary encoding

python3 scripts/mem_compress.py \x3Cworkspace> dict

Layer 3: Observation compression (session JSONL → summaries)

python3 scripts/mem_compress.py \x3Cworkspace> observe

Layer 5: Tokenizer optimization

python3 scripts/mem_compress.py \x3Cworkspace> optimize

Tiered summaries (L0/L1/L2)

python3 scripts/mem_compress.py \x3Cworkspace> tiers

Cross-file deduplication

python3 scripts/mem_compress.py \x3Cworkspace> dedup

Token count report

python3 scripts/mem_compress.py \x3Cworkspace> estimate

Workspace health check

python3 scripts/mem_compress.py \x3Cworkspace> audit

Global Options

--json Machine-readable JSON output --dry-run Preview without writing files --since DATE Filter sessions by date (YYYY-MM-DD) --auto-merge Auto-merge duplicates (dedup command)

Engram - Layer 6: Real-Time Observational Memory

Engram is the flagship layer. It operates as a live engine alongside conversations, automatically compressing messages into structured, priority-annotated knowledge.

Prerequisites

Configure via engram.yaml (recommended) or environment variables:


# engram.yaml - place in claw-compactor root
llm:
 provider: openai-compatible
 base_url: http://localhost:8403
 model: claude-code/sonnet
 max_tokens: 4096

threads:
 default:
 observer_threshold: 30000 # pending tokens before Observer fires
 reflector_threshold: 40000 # observation tokens before Reflector fires

concurrency:
 max_workers: 4 # parallel thread workers

# Alternative: environment variables
export ANTHROPIC_API_KEY=sk-ant-... # Preferred

# or
export OPENAI_API_KEY=sk-... # OpenAI-compatible fallback
export OPENAI_BASE_URL=https://... # Optional: custom endpoint (local LLM, etc.)

### Engram Auto-Mode (Recommended for Production)
Auto-detects all active threads and processes them concurrently (4 workers):

# Single run - auto-detects all threads
python3 scripts/engram_auto.py --workspace ~/.openclaw/workspace

# Via shell wrapper
bash scripts/engram-auto.sh

# Via CLI
python3 scripts/engram_cli.py \x3Cworkspace> auto --config engram.yaml
python3 scripts/engram_cli.py \x3Cworkspace> status --thread openclaw-main
python3 scripts/engram_cli.py \x3Cworkspace> observe --thread openclaw-main
python3 scripts/engram_cli.py \x3Cworkspace> reflect --thread openclaw-main

**Retry:** LLM calls retry on 429/5xx with exponential backoff (2s→4s→8s, max 3 attempts).
No retry on 400/401/403 (fail fast on config errors).

# Check all thread statuses
python3 scripts/mem_compress.py \x3Cworkspace> engram status

# Force Observer for a thread
python3 scripts/mem_compress.py \x3Cworkspace> engram observe --thread \x3Cthread-id>

# Force Reflector for a thread
python3 scripts/mem_compress.py \x3Cworkspace> engram reflect --thread \x3Cthread-id>

# Print injectable context
python3 scripts/mem_compress.py \x3Cworkspace> engram context --thread \x3Cthread-id>

# Status: all threads
python3 scripts/engram_cli.py \x3Cworkspace> status

# Status: single thread
python3 scripts/engram_cli.py \x3Cworkspace> status --thread \x3Cthread-id>

# Force observe
python3 scripts/engram_cli.py \x3Cworkspace> observe --thread \x3Cthread-id>

# Force reflect
python3 scripts/engram_cli.py \x3Cworkspace> reflect --thread \x3Cthread-id>

# Import conversation from file (JSON array or JSONL)
python3 scripts/engram_cli.py \x3Cworkspace> ingest \
 --thread \x3Cthread-id> --input /path/to/conversation.jsonl

# Get injectable context string (ready for system prompt)
python3 scripts/engram_cli.py \x3Cworkspace> context --thread \x3Cthread-id>

# JSON output for any command
python3 scripts/engram_cli.py \x3Cworkspace> status --json
python3 scripts/engram_cli.py \x3Cworkspace> context --thread \x3Cid> --json

# Start daemon, pipe JSONL messages via stdin
python3 scripts/engram_cli.py \x3Cworkspace> daemon --thread \x3Cthread-id>

# Pipe a message:
echo '{"role":"user","content":"Hello!","timestamp":"12:00"}' | \

# Control commands (send as JSONL):
echo '{"__cmd":"observe"}' # force observe now
echo '{"__cmd":"reflect"}' # force reflect now
echo '{"__cmd":"status"}' # print thread status JSON
echo '{"__cmd":"quit"}' # exit daemon

# Quiet mode (suppress startup messages on stderr)
python3 scripts/engram_cli.py \x3Cworkspace> daemon --thread \x3Cid> --quiet

### Engram Python API
```python
from scripts.lib.engram import EngramEngine

engine = EngramEngine(
 workspace_path="/path/to/workspace",
 observer_threshold=30_000, # tokens before auto-observe
 reflector_threshold=40_000, # tokens before auto-reflect
 anthropic_api_key="sk-ant-...", # or set ANTHROPIC_API_KEY env
)

# Add a message - auto-triggers observe/reflect when thresholds exceeded
status = engine.add_message("thread-id", role="user", content="Hello!")

# Manual trigger regardless of thresholds
obs_text = engine.observe("thread-id") # returns None if no pending msgs
ref_text = engine.reflect("thread-id") # returns None if no observations

# Get full context dict
ctx = engine.get_context("thread-id")

# Build injectable system context string
ctx_str = engine.build_system_context("thread-id")

# Ready to prepend to system prompt

### Engram Configuration Variables
`ANTHROPIC_API_KEY`, Default=-, Description=Anthropic API key (preferred)
`OPENAI_API_KEY`, Default=-, Description=OpenAI-compatible API key
`OPENAI_BASE_URL`, Default=`https://api.openai.com`, Description=Custom endpoint for local LLMs
`OM_OBSERVER_THRESHOLD`, Default=`30000`, Description=Pending tokens before auto-observe
`OM_REFLECTOR_THRESHOLD`, Default=`40000`, Description=Observation tokens before auto-reflect
`OM_MODEL`, Default=`claude-opus-4-5`, Description=LLM model override

### Threshold Tuning Quick Reference
Each Observer call ≈ 2K output tokens (Sonnet). Daily volume at default 30K threshold:

#aimm, Daily Tokens=~149K, @30K threshold=~5×/day, @10K threshold=~15×/day
openclaw-main, Daily Tokens=~138K, @30K threshold=~4.5×/day, @10K threshold=~14×/day
#open-compress, Daily Tokens=~68K, @30K threshold=~2.3×/day, @10K threshold=~7×/day
#general, Daily Tokens=~62K, @30K threshold=~2×/day, @10K threshold=~6×/day
subagent, Daily Tokens=~43K, @30K threshold=~1.4×/day, @10K threshold=~4×/day
cron, Daily Tokens=~9K, @30K threshold=~0.3×/day, @10K threshold=~1×/day
**Total**, Daily Tokens=**~470K/day**, @30K threshold=**~16×/day (~32K output tokens)**, @10K threshold=**~47×/day (~94K output tokens)**

Start at `observer_threshold: 30000`. Tune down for fresher context; tune up to reduce cost.

### Engram Benchmark Summary
| **Engram (L6)** | **87.5%** | 0.038 | 0.414 | ~35s | 2 |
| RuleCompressor (L1–5) | 9.0% | 0.923 | 0.958 | ~6ms | 0 |
| RandomDrop | 21.5% | 0.852 | 0.911 | ~0ms | 0 |

- Engram low ROUGE-L = semantic restructuring, not verbatim copy - intent is preserved
- Use RuleCompressor for instant prompt compression; Engram for long-term memory
- Full results → `benchmark/RESULTS.md`

### Observation Format
Engram produces structured, bilingual (EN/中文) priority-annotated logs:

Date: 2026-03-05
- 12:10 User building OpenCompress; deadline one week / 用户在构建 OpenCompress,deadline 一周内
 - 12:10 Using ModernBERT-large / 使用 ModernBERT-large
 - 🟡 12:12 Discussed annotation strategy / 讨论了标注策略
- 🟡 12:30 Deployment pipeline discussion on M3 Ultra
- 🟢 12:45 User prefers concise replies

- **Critical** - goals, deadlines, blockers, key decisions (never dropped)
- 🟡 **Important** - technical details, ongoing work, preferences
- 🟢 **Useful** - background, mentions, soft context

### Memory Storage Layout
memory/engram/{thread_id}/
├── pending.jsonl # Unobserved message buffer (auto-cleared after observe)
├── observations.md # Observer output - append-only structured log
├── reflections.md # Reflector output - compressed long-term memory (overwrites)
└── meta.json # Timestamps and token counts

## Integration with OpenClaw Memory System

### System Prompt Injection
Inject Engram context at the start of each session:

engine = EngramEngine(workspace_path)
ctx_str = engine.build_system_context("my-session")
if ctx_str:
 system_prompt = ctx_str + "\
\
" + base_system_prompt

The `build_system_context()` output structure:

## Long-Term Memory (Reflections)
\x3CReflector output - long-term compressed context>

## Recent Observations
\x3CLast 200 lines of Observer output>

\x3C!-- engram_tokens: 1234 -->

### Combining Engram with Deterministic Layers
After an Engram session, run the deterministic pipeline on the output files:

# Then apply deterministic compression to further reduce those:

### Recommended Workflow for Long-Running Agent Sessions
1. **Session start:** inject `build_system_context()` into system prompt
2. **Each message:** call `engine.add_message()` - auto-triggers observe/reflect
3. **Session end / weekly cron:** run `full` pipeline on workspace
4. **Multi-session continuity:** context persists in `memory/engram/{thread}/`

## OpenClaw Skill Installation
To install as an OpenClaw skill, ensure the skill directory is available at:
~/.openclaw/workspace/skills/claw-compactor/
or configure the path in your OpenClaw skill registry.

SKILL.md is read by the OpenClaw agent dispatcher. The `description` and
`triggers` fields above control when this skill is automatically activated.

## Heartbeat / Cron Automation
```markdown

## Memory Maintenance (weekly)
- python3 skills/claw-compactor/scripts/mem_compress.py \x3Cworkspace> benchmark
- If savings > 5%: run full pipeline
- If pending Engram messages: run engram observe --thread \x3Cid>

Cron (Sunday 3am):
0 3 * * 0 cd /path/to/skills/claw-compactor && \
 python3 scripts/mem_compress.py /path/to/workspace full

## Output Artifacts Reference
Dictionary codebook, Location=`memory/.codebook.json`, Description=Must travel with memory files
Observed session log, Location=`memory/.observed-sessions.json`, Description=Tracks processed transcripts
Layer 3 summaries, Location=`memory/observations/`, Description=Observation compression output
Engram observations, Location=`memory/engram/{thread}/observations.md`, Description=Live Observer log
Engram reflections, Location=`memory/engram/{thread}/reflections.md`, Description=Distilled long-term memory
Level 0 summary, Location=`memory/MEMORY-L0.md`, Description=~200 token ultra-compressed summary
Level 1 summary, Location=`memory/MEMORY-L1.md`, Description=~500 token compressed summary

## Troubleshooting
- `FileNotFoundError` on workspace: Point path to workspace root containing `memory/`
- Dictionary decompression fails: Check `memory/.codebook.json` is valid JSON
- Zero savings on `benchmark`: Workspace already optimized
- `observe` finds no transcripts: Check `sessions/` for `.jsonl` files
- Engram: "no API key configured": Set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`
- Engram Observer returns `None`: No pending messages for that thread
- Token counts seem wrong: Install tiktoken: `pip3 install tiktoken`

Usage Guidance

What to check before installing or running: - Do not run the skill automatically at session start. Disable any auto-run hooks and run only in a sandbox/VM first. - Audit the SKILL.md and code for network calls (look for http(s) requests, default base_url values) and for any places that read files like ~/.env, .env, or arbitrary filesystem paths. The benchmark code explicitly loads a .env file from the project root — remove or sanitize this if it could contain secrets. - The SKILL.md and scripts reference LLM API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY) but the skill metadata does not declare them. Treat LLM keys as sensitive: do not provide high-privilege keys to this skill until you confirm exactly which calls it will make and to which endpoints. - Search the repository for any code that sends data to external endpoints (HTTP POST/PUT), for hard-coded URLs/IPs, and for code that constructs 'injectable context' strings. If the skill prints or packages workspace content into a system-prompt-ready string, that can leak secrets or system prompts; ensure such output is sanitized before use. - If you must test it, run it on a non-sensitive dummy workspace and monitor outbound network traffic. Prefer ephemeral credentials and network isolation. - If the author/owner is unknown (metadata says 'unknown'), be more conservative: require provenance, an upstream repo link, or third-party review before trusting. What would change this assessment: - If the skill metadata explicitly declared required env vars and the registry enforced least-privilege (scoped, limited API keys), and the SKILL.md removed prompt-injection strings and unicode obfuscation, I would raise confidence toward benign. - If the owner is a known, reputable maintainer and there is an upstream GitHub repo with reproducible release artifacts and signed releases, that would reduce risk.

Capability Analysis

Type: OpenClaw Skill Name: claw-compactor Version: 1.0.0 The skill bundle provides a sophisticated 14-stage token compression pipeline and an LLM-driven memory system (Engram). However, it includes a complex Node.js proxy (proxy/server.mjs) designed to wrap the Claude CLI, which features high-risk capabilities such as spawning the CLI with the '--dangerously-skip-permissions' flag and providing an HTTP endpoint to kill processes by PID (/zombies POST). Furthermore, the proxy implements a significantly weak default authentication mechanism where the 'local-proxy' token grants full access to these sensitive functions by default. While these features are aligned with the stated goal of providing an autonomous agent gateway, the combination of process control, permission bypass, and poor default security configuration makes the bundle suspicious.

Capability Assessment

ℹ Purpose & Capability

Name/description (token compression + Engram LLM memory) align with the bundled code: a large compressor/engram implementation is present. However the skill package metadata declares no required env vars or credentials while the SKILL.md and code clearly expect LLM API keys and may call an LLM proxy (e.g., OPENAI_API_KEY/ANTHROPIC_API_KEY/OPENAI_BASE_URL). Also benchmark code auto-loads a .env file from the project root, which implies access to local secrets that was not declared.

⚠ Instruction Scope

SKILL.md instructs the agent to run scripts that read and compress an entire workspace (examples use ~/.openclaw/workspace), start daemons, ingest arbitrary files, and 'print injectable context' for system prompts. The code actively loads a .env file and supports ingesting arbitrary filesystem paths. Those behaviors go beyond a narrow helper and could expose secrets or large amounts of user data if run automatically.

ℹ Install Mechanism

There is no external install spec (no downloads), which lowers supply-chain risk. However the skill bundle includes 100+ source files that will be present on disk when the skill is installed; that means code will run from the skill directory and should be audited. No external archives or remote fetches were declared in the manifest.

⚠ Credentials

The registry metadata declares no required environment variables, but SKILL.md explicitly references ANTHROPIC_API_KEY, OPENAI_API_KEY, and OPENAI_BASE_URL and the code loads a .env file automatically. Requesting LLM API keys (and then not declaring them) is an incoherence and increases the risk that secrets could be read from the workspace or .env without the user's explicit consent.

ℹ Persistence & Privilege

The skill is not always:true and does not request elevated platform privileges. Autonomous invocation is permitted (default). That normal behavior combined with the instruction to 'run at session start' and the ability to process an entire workspace increases the impact if the skill is allowed to run without careful controls.

Version History

v1.0.0

Initial release of Claw Compactor: advanced 6-layer token compression for OpenClaw agents. - Reduces workspace token usage by 50–97% using layered, configurable compression (rule-engine, dictionary encoding, observation compression, RLE, protocol optimization, plus live LLM-based Engram memory). - Includes auto-mode for session-start savings reporting and batch workspace compression. - Multiple command-line utilities: compress, benchmark, report, estimate, deduplicate, audit, and more. - Engram system adds live, real-time observational memory, supporting concurrent threads and retry logic. - Full documentation for configuration (YAML/environment), CLI, and Python API usage.

Metadata

Slug claw-compactor

Version 1.0.0

License MIT-0

All-time Installs 2

Active Installs 2

Total Versions 1

Frequently Asked Questions

What is Claw Compactor?

Automation skill for Claw Compactor. It is an AI Agent Skill for Claude Code / OpenClaw, with 238 downloads so far.

How do I install Claw Compactor?

Run "/install claw-compactor" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.

Is Claw Compactor free?

Yes, Claw Compactor is completely free, licensed under MIT-0. You can download, install and use it at no cost.

Which platforms does Claw Compactor support?

Claw Compactor is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).

Who created Claw Compactor?

It is built and maintained by gandli-2025 (@gandli-2025); the current version is v1.0.0.

More Skills

Claw Compactor