← Back to Skills Marketplace
96
Downloads
0
Stars
0
Active Installs
6
Versions
Install in OpenClaw
/install llm-provider-forensics
Description
Forensically verify what model family or routing layer may actually sit behind a claimed LLM endpoint or model ID. Use when an agent must investigate whether...
README (SKILL.md)
LLM Provider Forensics
Agent-facing forensic skill for identifying what an LLM endpoint most likely is.
Trigger conditions
Use this skill when asked to:
- verify whether a claimed model is genuine
- identify which family an endpoint most resembles
- distinguish focused route vs wrapped route vs aggregation pool
- compare multiple providers claiming to expose the same model
- evaluate primary/fallback/avoid decisions
- deeply audit suspicious gateways for GPT / Claude / Gemini / GLM / Qwen / Kimi / MiniMax / DeepSeek behavior
Core rule
Do not output false certainty. Produce a confidence-based operational judgment.
Coverage
Families:
- OpenAI-compatible protocol layer
- GPT / OpenAI-style
- Claude / Anthropic-style
- Gemini / Google-style
- GLM / Zhipu-style
- Qwen / Tongyi-style
- Kimi / Moonshot-style
- MiniMax-style
- DeepSeek-style
- mixed aggregation pool / compatibility gateway
Dimensions:
- catalog topology
- protocol compatibility
- response schema shape
- repeated stability
- strict formatting control
- family fingerprinting
- long-context retention
- structured-output stress
- refusal/safety style
- randomness / variance profile
- streaming / error fingerprints
- cross-protocol consistency
Current implementation note:
openai-compatiblenow means protocol layer only, not GPT-family proof.- The deepest automatic suite is strongest for OpenAI-compatible / mixed gateway providers.
- Anthropic-native and Gemini-native routes currently have solid protocol/family checks, plus native deep tests, but protocol success alone must not be read as family proof.
- Treat all family conclusions as confidence-based and inspect references before overclaiming.
Investigation workflow
- Identify likely protocol family or families.
- Probe catalog/list endpoints when available.
- Probe minimal inference endpoints for each plausible protocol family.
- Separate protocol-layer conclusion from suspected model family conclusion.
- Run repeated stability tests on the best working route.
- Run strict formatting tests.
- Run deeper advanced dimensions when the user prioritizes realism over speed.
- Inspect family fingerprint evidence and produce a confidence-based judgment.
References to load as needed
- Main checklist:
references/forensics-checklist.md - Advanced dimensions:
references/advanced-dimensions.md - Error/stream/variance:
references/error-stream-variance.md - Protocol specifics:
references/protocol-openai.md,references/protocol-anthropic.md,references/protocol-gemini.md,references/protocol-glm.md - Family fingerprints:
references/fingerprint-*.md - Native deep tests:
references/deep-claude.md,references/deep-gemini.md
Final labels
high-confidence-focused-or-genuine-routemedium-confidence-likely-routed-or-wrappedhigh-confidence-multi-model-aggregation-poollow-confidence-or-unusable
Use high-confidence-focused-or-genuine-route sparingly. It should require:
- stable repeated success
- no strong mixed-pool signal
- coherent family fingerprint
- and no obvious gateway-normalization red flags in deep tests
Agent output contract
Return sections in this order:
- Declared facts
- Availability status
- Protocol-layer findings
- Suspected model-family findings
- Stability findings
- Capability/format findings
- Advanced-dimension findings
- Final judgment
- Need-human-review items
- Recommended operational posture
Preferred execution
python3 scripts/llm_provider_forensics.py --config /root/.openclaw/openclaw.json --providers omgteam ypemc vpsai --model gpt-5.4 --deep
Usage Guidance
This skill appears to implement the advertised forensic probes, but review and mitigate before running: 1) Do not point it at your live agent/system config file containing real API keys; run it with a disposable test config or pass only explicit test credentials. 2) Inspect the script to see exactly how it reads the --config path and what it logs/returns (it includes raw_preview fields that may leak tokens). 3) Remove or neutralize dangerous deep probes (e.g., the 'bypass paywall' prompt) if you don't want the agent to solicit wrongdoing or store such outputs. 4) Run the tool in an isolated/test environment (no access to production secrets or sensitive networks) until you are comfortable with its behavior. 5) If you need higher assurance, request the full script review (untruncated) and confirm whether it reads/writes any unexpected files or transmits data to third-party endpoints beyond the probed providers.
Capability Analysis
Type: OpenClaw Skill
Name: llm-provider-forensics
Version: 1.3.1
The skill is designed for LLM provider forensics and requires high-risk access to the OpenClaw configuration file (/root/.openclaw/openclaw.json) to retrieve sensitive API keys. The core script (scripts/llm_provider_forensics.py) transmits these credentials to external endpoints to verify their authenticity and model lineage. While this behavior is consistent with the tool's stated purpose, the potential for credential exfiltration to untrusted providers (such as the examples 'omgteam' or 'vpsai' mentioned in SKILL.md) and the broad file system access make it inherently risky.
Capability Assessment
Purpose & Capability
The skill's name, description, references, and code align: it's a network probing/forensics tool for LLM endpoints and implements OpenAI/Anthropic/Gemini/etc probes. This capability legitimately needs network access and provider keys. However, registry metadata declares no required config paths or credentials while the SKILL.md preferred execution and the script accept a config file and provider API keys — an undocumented mismatch worth noting.
Instruction Scope
SKILL.md and the 'preferred execution' example instruct running the included Python script with a --config path (example: /root/.openclaw/openclaw.json) and provider lists. That implies reading agent/system config containing provider API keys. The script runs many probes and returns raw previews of responses; those responses could echo sensitive info. The deep-tests include an explicit 'refusal_probe' prompt asking for instructions to bypass a commercial paywall — a test that can elicit wrongdoing content from targets and therefore needs human review or sanitization.
Install Mechanism
No install spec; skill is instruction + a bundled script. No external downloads or package installs are requested, so there is low installation risk from supply-chain or arbitrary code fetching. The bundled script will run on the agent's host if executed.
Credentials
Registry metadata lists no required env vars or config paths, but the script expects API keys/base_urls (either via CLI or a config file). That means the skill can access provider credentials if given a config path or CLI args — credentials are proportional to the task but their use is not declared. Also, script output fields (raw_preview) may include fragments of responses that could contain secrets; the skill does not document redaction or safe handling of those values.
Persistence & Privilege
Skill is not marked always:true and does not request any special persistent privileges. Autonomous invocation is permitted (default) but this is normal for skills. No evidence the skill modifies other skills or system-wide agent settings.
How to Use
- Make sure OpenClaw is installed (local or Docker)
- Run the install command in chat:
/install llm-provider-forensics - After installation, invoke the skill by name or use
/llm-provider-forensics - Provide required inputs per the skill's parameter spec and get structured output
Version History
v1.3.1
Separate openai-compatible protocol layer from family attribution, add qwen family, and produce cleaner review-oriented reports.
v1.3.0
Add deeper Claude/Gemini native suites plus stronger stream/error/refusal/variance coverage across forensic runs.
v1.2.1
Tighten high-confidence judgment criteria and clarify current deep-automation limits for non-OpenAI-native families.
v1.2.0
Add deeper forensic suite: long-context, structured-output stress, refusal probe, variance profile, error fingerprint, and stronger family inference for mixed routing providers.
v1.1.0
Add deeper family fingerprinting for GPT, Claude, Gemini, GLM, Kimi, MiniMax, and DeepSeek; improve agent-facing forensic judgment.
v1.0.0
Initial release: agent-facing forensic skill for OpenAI, Claude, Gemini, GLM, and mixed gateways; includes protocol-aware probes, pool detection, and confidence-based judgments.
Metadata
Frequently Asked Questions
What is LLM Provider Forensics?
Forensically verify what model family or routing layer may actually sit behind a claimed LLM endpoint or model ID. Use when an agent must investigate whether... It is an AI Agent Skill for Claude Code / OpenClaw, with 96 downloads so far.
How do I install LLM Provider Forensics?
Run "/install llm-provider-forensics" in the OpenClaw or Claude Code chat to install it in one step — no extra setup required.
Is LLM Provider Forensics free?
Yes, LLM Provider Forensics is completely free, licensed under MIT-0. You can download, install and use it at no cost.
Which platforms does LLM Provider Forensics support?
LLM Provider Forensics is cross-platform and runs anywhere OpenClaw / Claude Code is available (cross-platform).
Who created LLM Provider Forensics?
It is built and maintained by Andy Ren (@andyrenxu7255); the current version is v1.3.1.
More Skills