Genai Toolkit

Genai Toolkit v2.0.0 — an AI toolkit for managing generative AI workflows from the command line. Log configurations, benchmarks, prompts, evaluations, fine-tuning runs, cost tracking, and optimization notes. Each entry is timestamped and persisted locally. Works entirely offline — your data never leaves your machine.

Why Genai Toolkit?

• Works entirely offline — your data never leaves your machine
• Simple command-line interface with no GUI dependency
• Export to JSON, CSV, or plain text at any time for sharing or archival
• Automatic activity history logging across all commands
• Each domain command doubles as both a logger and a viewer

Commands

Domain Commands

Each domain command works in two modes: log mode (with arguments) saves a timestamped entry, view mode (no arguments) shows the 20 most recent entries.

Utility Commands

Data Storage

All data is stored locally at ~/.local/share/genai-toolkit/. Each domain command writes to its own log file (e.g., configure.log, benchmark.log). A unified history.log tracks all actions across commands. Use export to back up your data at any time.

Requirements

• Bash (4.0+)
• No external dependencies — pure shell script
• No network access required

When to Use

• Tracking AI model benchmarks and comparisons across different providers and versions over time
• Logging prompt engineering iterations to understand what improvements actually moved the needle
• Monitoring API costs and token usage across multiple projects and billing periods
• Evaluating fine-tuning experiments with detailed hyperparameter and metric tracking
• Building a searchable knowledge base of optimization attempts and analysis insights

Examples

# Log a benchmark result
genai-toolkit benchmark "GPT-4o latency: avg 1.2s, p99 3.8s on summarization task, 500 samples"

# Track a cost entry
genai-toolkit cost "March batch processing: $42.50 across 15k requests, avg $0.0028/req"

# Compare two models
genai-toolkit compare "Claude 3.5 vs GPT-4o on code generation — Claude 15% faster, GPT-4o 5% more accurate"

# Log a prompt iteration
genai-toolkit prompt "v3: Added chain-of-thought instruction, reduced hallucination rate from 12% to 3%"

# Record a fine-tuning run
genai-toolkit fine-tune "SQL-gen model epoch 5: accuracy=0.96, loss=0.12, lr=2e-5, dataset=50k rows"

# View all statistics
genai-toolkit stats

# Export everything to JSON
genai-toolkit export json

# Search for entries mentioning latency
genai-toolkit search latency

# Check recent activity
genai-toolkit recent

# Health check
genai-toolkit status

Powered by BytesAgain | bytesagain.com | [email protected]