← Back to Skills Marketplace
legiovi

Tokenizer

by legiovi · GitHub ↗ · v1.0.2 · MIT-0
cross-platform ✓ Security Clean
96
Downloads
1
Stars
0
Active Installs
2
Versions
Install in OpenClaw
/install token-optimizer-skills
Description
Full token economy suite for OpenClaw agents. Audits context window usage (skills, history, tool outputs), then applies 5 creative strategies to reduce bloat...
Usage Guidance
This package appears to do what it says: count tokens, analyze skill metadata, distill conversation history, and optionally compress large documents. Before installing, review these points: - The tools read local skill files and conversation history (they look in ~/.openclaw, /app/skills, and other candidate paths). If your system prompt, skills, or chat logs contain sensitive data, be aware this skill will access them during audits. - Distillation writes JSON files to an episodic store (by default .openclaw/memory/episodic). Confirm that location and retention policy are acceptable for your data. - compress_prompt.py requires an external dependency (llmlingua) and is guarded OFFLINE_ONLY in the config; do not run compression against live system prompts or code (the manifest and skill_runner enforce this guardrail, but double-check in your deployment). - No credentials are requested, and there are no remote-download install steps in the manifest — still, verify the source of this skill bundle (owner and homepage are missing) before trusting it with production data. - If you plan to enable automated distillation (memory_agent.autonomous), test in a contained environment first so you understand when and where histories are archived or flushed. If you want higher assurance, ask the publisher for a provenance record or validate the full, untruncated source files locally before enabling the skill in an agent that handles sensitive content.
Capability Analysis
Type: OpenClaw Skill Name: token-optimizer-skills Version: 1.0.2 The 'token-economy' skill bundle is a legitimate and well-documented suite for managing LLM context tokens. It includes functional scripts for token counting (count_tokens.py), skill metadata analysis (analyze_skills.py), and conversation distillation (distill_memory.py). The bundle uses a structured orchestration approach with a central configuration (orchestrator_config.json) and a dispatcher (skill_runner.py) to execute bundled Python scripts. While the scripts have the capability to read local files, this behavior is essential for their stated purpose of auditing and optimizing context. No evidence of malicious intent, data exfiltration, or harmful prompt injection was found; in fact, the documentation includes explicit safety guardrails for high-risk operations like prompt compression.
Capability Assessment
Purpose & Capability
Name/description (token audit & optimization) align with the provided scripts: count_tokens.py, analyze_skills.py, distill_memory.py, compress_prompt.py and orchestration helpers. The scripts perform token counting, skill-metadata analysis, memory distillation and optional prompt compression — all expected for a token-economy skill.
Instruction Scope
SKILL.md explicitly instructs the agent to read loaded skills' metadata, conversation history, and tool outputs. The included scripts implement that: analyze_skills.py scans SKILL.md files in several candidate skill directories and count_tokens.py/distill_memory.py operate on history files provided. This is coherent with the purpose but means the skill will read system/skill files and conversation history (potentially exposing system prompts or other sensitive context) — the README and scripts do include guardrails (e.g., OFFLINE_ONLY for compress_prompt.py).
Install Mechanism
No install specification is provided (instruction-only skill with bundled scripts). Optional Python dependencies are documented (tiktoken, transformers, llmlingua). No remote downloads or extraction steps are present in the manifest; code is local and runs via subprocess, which is appropriate and low-risk for this sort of utility.
Credentials
The skill declares no required environment variables or credentials. It optionally respects a SKILLS_DIR env var for locating skill files; other file-paths are local/defaults. No secrets, tokens, or unrelated cloud credentials are requested.
Persistence & Privilege
The orchestrator_config.json and memory_agent.py are designed to write distilled memory into an episodic store (default path .openclaw/memory/episodic) and memory_agent is marked 'autonomous' in the bundled config. The skill is not marked always:true in registry metadata, but the included config and scripts do enable background distillation behavior if an orchestrator wires them up. This is functional for the skill's purpose but is a persistence/automation behavior you should review before enabling in production.
How to Use
  1. Make sure OpenClaw is installed (local or Docker)
  2. Run the install command in chat: /install token-optimizer-skills
  3. After installation, invoke the skill by name or use /token-optimizer-skills
  4. Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.0.2
### Initial orchestration layer added for token-optimizer-skills. - Introduced four new files: `memory_agent.py`, `orchestrator_config.json`, `router.py`, and `skill_runner.py`. - Lays the foundation for modular and automated skill orchestration within the token optimization workflow. - No existing files modified; these additions prepare the skill for more advanced, multi-agent or multi-skill execution patterns.
v1.0.0
Most OpenClaw agents suffer from context window bloat as conversations grow. Loaded skills, long histories, large tool outputs, and repetitive code quickly eat up tokens — leading to higher costs, slower responses, truncated context, and degraded performance. Token Economy Skill Suite fixes this by giving your agent built-in intelligence to monitor and actively manage its own token usage. Key Changes It Introduces: Proactive Diagnosis — The token-audit skill analyzes the current context and produces a professional Token Budget Report showing exactly where tokens are being wasted (skills metadata, history, tool dumps, etc.). Aggressive Optimization — The token-optimizer skill applies 5 powerful strategies: Memory Distillation — Converts long chat histories into compact structured JSON facts (decisions, preferences, constraints, next actions) — often saving 40–70% tokens. Prompt Compression — Uses LLMLingua-2 to shrink large documents or RAG results while keeping meaning intact. Skill Lazy Loading — Identifies and flags bloated SKILL.md descriptions so you can keep only what you need. Context DNA Compression — Summarizes repetitive boilerplate code (imports, UI layouts, standard patterns) into tiny comments — up to 80% savings on code-heavy tasks. Model Dialect Rewriting — Rewrites prompts in the most token-efficient format for your model (especially XML-style for Gemma). Persistent Memory Hygiene — Encourages moving data from expensive Working Memory to cheap Episodic (JSON facts) and Semantic stores, preventing the context death spiral in long-running projects. Real-World Impact: Dramatically lower token usage and API costs in extended sessions More reliable agent behavior (less truncation, better recall of past decisions) Cleaner, faster context windows Works model-agnostically but includes Gemma-specific optimizations Includes ready-to-run helper scripts (analyze_skills.py, distill_memory.py, compress_prompt.py, count_tokens.py) In short: This skill transforms your agent from a passive token burner into a self-optimizing, token-aware system — essential for anyone doing serious work with OpenClaw over long conversations or large codebases.
Metadata
Slug token-optimizer-skills
Version 1.0.2
License MIT-0
All-time Installs 0
Active Installs 0
Total Versions 2
Frequently Asked Questions

What is Tokenizer?

Full token economy suite for OpenClaw agents. Audits context window usage (skills, history, tool outputs), then applies 5 creative strategies to reduce bloat... It is an AI Agent Skill for Claude Code / OpenClaw, with 96 downloads so far.

How do I install Tokenizer?

Run "/install token-optimizer-skills" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.

Is Tokenizer free?

Yes, Tokenizer is completely free, licensed under MIT-0. You can download, install and use it at no cost.

Which platforms does Tokenizer support?

Tokenizer is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).

Who created Tokenizer?

It is built and maintained by legiovi (@legiovi); the current version is v1.0.2.

💬 Comments