Datasets

A data processing toolkit for ingesting, transforming, querying, and managing dataset entries from the command line. All operations are logged with timestamps and stored locally.

Commands

Data Operations

Each data command works in two modes: run without arguments to view recent entries, or pass input to record a new entry.

Utility Commands

Data Storage

All data is stored locally at ~/.local/share/datasets/:

• Each data command writes to its own log file (e.g., ingest.log, transform.log)
• Entries are stored as timestamp|value pairs (pipe-delimited)
• All actions are tracked in history.log with timestamps
• Export generates files in the data directory (export.json, export.csv, or export.txt)

Requirements

• Bash (with set -euo pipefail)
• Standard Unix utilities: date, wc, du, grep, tail, cat, sed
• No external dependencies or API keys required

When to Use

• To log and track data processing operations (ingest, transform, query, etc.)
• To maintain a searchable history of data pipeline activities
• To export accumulated records in JSON, CSV, or plain text format
• As part of larger automation or data-pipeline workflows
• When you need a lightweight, local-only dataset operation tracker

Examples

# Record a new ingest entry
datasets ingest "loaded training_data.csv 10000 rows"

# View recent transform entries
datasets transform

# Record a query
datasets query "filter by date > 2026-01-01"

# Search across all logs
datasets search "training"

# Export everything as JSON
datasets export json

# Check overall statistics
datasets stats

# View recent activity
datasets recent

# Health check
datasets status

Powered by BytesAgain | bytesagain.com | [email protected] 💬 Feedback & Feature Requests: https://bytesagain.com/feedback