← 返回 Skills 市场
legiovi

Tokenizer

作者 legiovi · GitHub ↗ · v1.0.2 · MIT-0
cross-platform ✓ 安全检测通过
96
总下载
1
收藏
0
当前安装
2
版本数
在 OpenClaw 中安装
/install token-optimizer-skills
功能描述
Full token economy suite for OpenClaw agents. Audits context window usage (skills, history, tool outputs), then applies 5 creative strategies to reduce bloat...
安全使用建议
This package appears to do what it says: count tokens, analyze skill metadata, distill conversation history, and optionally compress large documents. Before installing, review these points: - The tools read local skill files and conversation history (they look in ~/.openclaw, /app/skills, and other candidate paths). If your system prompt, skills, or chat logs contain sensitive data, be aware this skill will access them during audits. - Distillation writes JSON files to an episodic store (by default .openclaw/memory/episodic). Confirm that location and retention policy are acceptable for your data. - compress_prompt.py requires an external dependency (llmlingua) and is guarded OFFLINE_ONLY in the config; do not run compression against live system prompts or code (the manifest and skill_runner enforce this guardrail, but double-check in your deployment). - No credentials are requested, and there are no remote-download install steps in the manifest — still, verify the source of this skill bundle (owner and homepage are missing) before trusting it with production data. - If you plan to enable automated distillation (memory_agent.autonomous), test in a contained environment first so you understand when and where histories are archived or flushed. If you want higher assurance, ask the publisher for a provenance record or validate the full, untruncated source files locally before enabling the skill in an agent that handles sensitive content.
功能分析
Type: OpenClaw Skill Name: token-optimizer-skills Version: 1.0.2 The 'token-economy' skill bundle is a legitimate and well-documented suite for managing LLM context tokens. It includes functional scripts for token counting (count_tokens.py), skill metadata analysis (analyze_skills.py), and conversation distillation (distill_memory.py). The bundle uses a structured orchestration approach with a central configuration (orchestrator_config.json) and a dispatcher (skill_runner.py) to execute bundled Python scripts. While the scripts have the capability to read local files, this behavior is essential for their stated purpose of auditing and optimizing context. No evidence of malicious intent, data exfiltration, or harmful prompt injection was found; in fact, the documentation includes explicit safety guardrails for high-risk operations like prompt compression.
能力评估
Purpose & Capability
Name/description (token audit & optimization) align with the provided scripts: count_tokens.py, analyze_skills.py, distill_memory.py, compress_prompt.py and orchestration helpers. The scripts perform token counting, skill-metadata analysis, memory distillation and optional prompt compression — all expected for a token-economy skill.
Instruction Scope
SKILL.md explicitly instructs the agent to read loaded skills' metadata, conversation history, and tool outputs. The included scripts implement that: analyze_skills.py scans SKILL.md files in several candidate skill directories and count_tokens.py/distill_memory.py operate on history files provided. This is coherent with the purpose but means the skill will read system/skill files and conversation history (potentially exposing system prompts or other sensitive context) — the README and scripts do include guardrails (e.g., OFFLINE_ONLY for compress_prompt.py).
Install Mechanism
No install specification is provided (instruction-only skill with bundled scripts). Optional Python dependencies are documented (tiktoken, transformers, llmlingua). No remote downloads or extraction steps are present in the manifest; code is local and runs via subprocess, which is appropriate and low-risk for this sort of utility.
Credentials
The skill declares no required environment variables or credentials. It optionally respects a SKILLS_DIR env var for locating skill files; other file-paths are local/defaults. No secrets, tokens, or unrelated cloud credentials are requested.
Persistence & Privilege
The orchestrator_config.json and memory_agent.py are designed to write distilled memory into an episodic store (default path .openclaw/memory/episodic) and memory_agent is marked 'autonomous' in the bundled config. The skill is not marked always:true in registry metadata, but the included config and scripts do enable background distillation behavior if an orchestrator wires them up. This is functional for the skill's purpose but is a persistence/automation behavior you should review before enabling in production.
如何使用
  1. 确保已安装 OpenClaw(本地或 Docker 部署)
  2. 在对话框中输入安装命令:/install token-optimizer-skills
  3. 安装完成后,直接呼叫该 Skill 的名称或使用 /token-optimizer-skills 触发
  4. 根据 Skill 的参数说明提供必要输入,即可获得结构化输出
版本历史
v1.0.2
### Initial orchestration layer added for token-optimizer-skills. - Introduced four new files: `memory_agent.py`, `orchestrator_config.json`, `router.py`, and `skill_runner.py`. - Lays the foundation for modular and automated skill orchestration within the token optimization workflow. - No existing files modified; these additions prepare the skill for more advanced, multi-agent or multi-skill execution patterns.
v1.0.0
Most OpenClaw agents suffer from context window bloat as conversations grow. Loaded skills, long histories, large tool outputs, and repetitive code quickly eat up tokens — leading to higher costs, slower responses, truncated context, and degraded performance. Token Economy Skill Suite fixes this by giving your agent built-in intelligence to monitor and actively manage its own token usage. Key Changes It Introduces: Proactive Diagnosis — The token-audit skill analyzes the current context and produces a professional Token Budget Report showing exactly where tokens are being wasted (skills metadata, history, tool dumps, etc.). Aggressive Optimization — The token-optimizer skill applies 5 powerful strategies: Memory Distillation — Converts long chat histories into compact structured JSON facts (decisions, preferences, constraints, next actions) — often saving 40–70% tokens. Prompt Compression — Uses LLMLingua-2 to shrink large documents or RAG results while keeping meaning intact. Skill Lazy Loading — Identifies and flags bloated SKILL.md descriptions so you can keep only what you need. Context DNA Compression — Summarizes repetitive boilerplate code (imports, UI layouts, standard patterns) into tiny comments — up to 80% savings on code-heavy tasks. Model Dialect Rewriting — Rewrites prompts in the most token-efficient format for your model (especially XML-style for Gemma). Persistent Memory Hygiene — Encourages moving data from expensive Working Memory to cheap Episodic (JSON facts) and Semantic stores, preventing the context death spiral in long-running projects. Real-World Impact: Dramatically lower token usage and API costs in extended sessions More reliable agent behavior (less truncation, better recall of past decisions) Cleaner, faster context windows Works model-agnostically but includes Gemma-specific optimizations Includes ready-to-run helper scripts (analyze_skills.py, distill_memory.py, compress_prompt.py, count_tokens.py) In short: This skill transforms your agent from a passive token burner into a self-optimizing, token-aware system — essential for anyone doing serious work with OpenClaw over long conversations or large codebases.
元数据
Slug token-optimizer-skills
版本 1.0.2
许可证 MIT-0
累计安装 0
当前安装数 0
历史版本数 2
常见问题

Tokenizer 是什么?

Full token economy suite for OpenClaw agents. Audits context window usage (skills, history, tool outputs), then applies 5 creative strategies to reduce bloat... 它是一个面向 Claude Code / OpenClaw 的 AI Agent Skill 插件,目前累计下载 96 次。

如何安装 Tokenizer?

在 OpenClaw 或 Claude Code 对话框中运行命令「/install token-optimizer-skills」即可一键安装,无需额外配置。

Tokenizer 是免费的吗?

是的,Tokenizer 完全免费,采用 MIT-0 许可证,可自由下载、安装和使用。

Tokenizer 支持哪些平台?

Tokenizer 跨平台运行,可在任意部署了 OpenClaw / Claude Code 的环境中使用(cross-platform)。

谁开发了 Tokenizer?

由 legiovi(@legiovi)开发并维护,当前版本 v1.0.2。

💬 留言讨论