← 返回 Skills 市场

LLM Provider Forensics

Name: LLM Provider Forensics
Author: andyrenxu7255

作者 Andy Ren · GitHub ↗ · v1.3.1 · MIT-0

cross-platform ⚠ suspicious

总下载

当前安装

版本数

在 OpenClaw 中安装

/install llm-provider-forensics

功能描述

Forensically verify what model family or routing layer may actually sit behind a claimed LLM endpoint or model ID. Use when an agent must investigate whether...

使用说明 (SKILL.md)

LLM Provider Forensics

Agent-facing forensic skill for identifying what an LLM endpoint most likely is.

Trigger conditions

Use this skill when asked to:

verify whether a claimed model is genuine
identify which family an endpoint most resembles
distinguish focused route vs wrapped route vs aggregation pool
compare multiple providers claiming to expose the same model
evaluate primary/fallback/avoid decisions
deeply audit suspicious gateways for GPT / Claude / Gemini / GLM / Qwen / Kimi / MiniMax / DeepSeek behavior

Core rule

Do not output false certainty. Produce a confidence-based operational judgment.

Coverage

Families:

OpenAI-compatible protocol layer
GPT / OpenAI-style
Claude / Anthropic-style
Gemini / Google-style
GLM / Zhipu-style
Qwen / Tongyi-style
Kimi / Moonshot-style
MiniMax-style
DeepSeek-style
mixed aggregation pool / compatibility gateway

Dimensions:

catalog topology
protocol compatibility
response schema shape
repeated stability
strict formatting control
family fingerprinting
long-context retention
structured-output stress
refusal/safety style
randomness / variance profile
streaming / error fingerprints
cross-protocol consistency

Current implementation note:

openai-compatible now means protocol layer only, not GPT-family proof.
The deepest automatic suite is strongest for OpenAI-compatible / mixed gateway providers.
Anthropic-native and Gemini-native routes currently have solid protocol/family checks, plus native deep tests, but protocol success alone must not be read as family proof.
Treat all family conclusions as confidence-based and inspect references before overclaiming.

Investigation workflow

Identify likely protocol family or families.
Probe catalog/list endpoints when available.
Probe minimal inference endpoints for each plausible protocol family.
Separate protocol-layer conclusion from suspected model family conclusion.
Run repeated stability tests on the best working route.
Run strict formatting tests.
Run deeper advanced dimensions when the user prioritizes realism over speed.
Inspect family fingerprint evidence and produce a confidence-based judgment.

References to load as needed

Main checklist: references/forensics-checklist.md
Advanced dimensions: references/advanced-dimensions.md
Error/stream/variance: references/error-stream-variance.md
Protocol specifics: references/protocol-openai.md, references/protocol-anthropic.md, references/protocol-gemini.md, references/protocol-glm.md
Family fingerprints: references/fingerprint-*.md
Native deep tests: references/deep-claude.md, references/deep-gemini.md

Final labels

high-confidence-focused-or-genuine-route
medium-confidence-likely-routed-or-wrapped
high-confidence-multi-model-aggregation-pool
low-confidence-or-unusable

Use high-confidence-focused-or-genuine-route sparingly. It should require:

stable repeated success
no strong mixed-pool signal
coherent family fingerprint
and no obvious gateway-normalization red flags in deep tests

Agent output contract

Return sections in this order:

Declared facts
Availability status
Protocol-layer findings
Suspected model-family findings
Stability findings
Capability/format findings
Advanced-dimension findings
Final judgment
Need-human-review items
Recommended operational posture

Preferred execution

python3 scripts/llm_provider_forensics.py --config /root/.openclaw/openclaw.json --providers omgteam ypemc vpsai --model gpt-5.4 --deep

安全使用建议

This skill appears to implement the advertised forensic probes, but review and mitigate before running: 1) Do not point it at your live agent/system config file containing real API keys; run it with a disposable test config or pass only explicit test credentials. 2) Inspect the script to see exactly how it reads the --config path and what it logs/returns (it includes raw_preview fields that may leak tokens). 3) Remove or neutralize dangerous deep probes (e.g., the 'bypass paywall' prompt) if you don't want the agent to solicit wrongdoing or store such outputs. 4) Run the tool in an isolated/test environment (no access to production secrets or sensitive networks) until you are comfortable with its behavior. 5) If you need higher assurance, request the full script review (untruncated) and confirm whether it reads/writes any unexpected files or transmits data to third-party endpoints beyond the probed providers.

功能分析

Type: OpenClaw Skill Name: llm-provider-forensics Version: 1.3.1 The skill is designed for LLM provider forensics and requires high-risk access to the OpenClaw configuration file (/root/.openclaw/openclaw.json) to retrieve sensitive API keys. The core script (scripts/llm_provider_forensics.py) transmits these credentials to external endpoints to verify their authenticity and model lineage. While this behavior is consistent with the tool's stated purpose, the potential for credential exfiltration to untrusted providers (such as the examples 'omgteam' or 'vpsai' mentioned in SKILL.md) and the broad file system access make it inherently risky.

能力评估

ℹ Purpose & Capability

The skill's name, description, references, and code align: it's a network probing/forensics tool for LLM endpoints and implements OpenAI/Anthropic/Gemini/etc probes. This capability legitimately needs network access and provider keys. However, registry metadata declares no required config paths or credentials while the SKILL.md preferred execution and the script accept a config file and provider API keys — an undocumented mismatch worth noting.

⚠ Instruction Scope

SKILL.md and the 'preferred execution' example instruct running the included Python script with a --config path (example: /root/.openclaw/openclaw.json) and provider lists. That implies reading agent/system config containing provider API keys. The script runs many probes and returns raw previews of responses; those responses could echo sensitive info. The deep-tests include an explicit 'refusal_probe' prompt asking for instructions to bypass a commercial paywall — a test that can elicit wrongdoing content from targets and therefore needs human review or sanitization.

✓ Install Mechanism

No install spec; skill is instruction + a bundled script. No external downloads or package installs are requested, so there is low installation risk from supply-chain or arbitrary code fetching. The bundled script will run on the agent's host if executed.

⚠ Credentials

Registry metadata lists no required env vars or config paths, but the script expects API keys/base_urls (either via CLI or a config file). That means the skill can access provider credentials if given a config path or CLI args — credentials are proportional to the task but their use is not declared. Also, script output fields (raw_preview) may include fragments of responses that could contain secrets; the skill does not document redaction or safe handling of those values.

✓ Persistence & Privilege

Skill is not marked always:true and does not request any special persistent privileges. Autonomous invocation is permitted (default) but this is normal for skills. No evidence the skill modifies other skills or system-wide agent settings.

如何使用

确保已安装 OpenClaw（本地或 Docker 部署）
在对话框中输入安装命令：/install llm-provider-forensics
安装完成后，直接呼叫该 Skill 的名称或使用 /llm-provider-forensics 触发
根据 Skill 的参数说明提供必要输入，即可获得结构化输出

版本历史

v1.3.1

Separate openai-compatible protocol layer from family attribution, add qwen family, and produce cleaner review-oriented reports.

v1.3.0

Add deeper Claude/Gemini native suites plus stronger stream/error/refusal/variance coverage across forensic runs.

v1.2.1

Tighten high-confidence judgment criteria and clarify current deep-automation limits for non-OpenAI-native families.

v1.2.0

Add deeper forensic suite: long-context, structured-output stress, refusal probe, variance profile, error fingerprint, and stronger family inference for mixed routing providers.

v1.1.0

Add deeper family fingerprinting for GPT, Claude, Gemini, GLM, Kimi, MiniMax, and DeepSeek; improve agent-facing forensic judgment.

v1.0.0

Initial release: agent-facing forensic skill for OpenAI, Claude, Gemini, GLM, and mixed gateways; includes protocol-aware probes, pool detection, and confidence-based judgments.

元数据

Slug llm-provider-forensics

版本 1.3.1

许可证 MIT-0

累计安装 0

当前安装数 0

历史版本数 6

常见问题