clean-http-toolkit

v0.1.0

Fifth member of the clean-* family. AI agents constantly need to fetch a URL, POST to an API, download a dataset, or run a mock server for tests — and currently the only option is shell-out to curl (no retry logic) or pip-install requests. This toolkit gives them all four in pure stdlib.

Scripts

• scripts/get.py — HTTP GET with retries (5xx/429), gzip decoding, redirect-following, --bearer TOKEN, --basic USER:PASS, --header 'Name: V', --query K=V, --json pretty-print, --print-headers, --status-only, --fail (exit 1 on non-2xx).
• scripts/post.py — POST / PUT / PATCH / DELETE via --method. Body sources: --json '\x3CJSON>', --json-file PATH, --form K=V (repeatable), --raw-file PATH (with --content-type). Same auth + retry options as get.py.
• scripts/download.py — streaming file download with progress bar on stderr, retries on transient errors, --resume for incomplete files, --sha256 HEX / --md5 HEX verification, configurable --chunk-size. Memory stays bounded regardless of file size.
• scripts/serve.py — tiny local HTTP server for tests. Two modes: static (serve a directory, like python3 -m http.server but with --max-requests N cutoff so CI tests can run and exit cleanly) and echo (reply to every request with a JSON envelope describing the request itself — perfect for webhook smoke tests). Binds to 127.0.0.1 by default for safety.
• scripts/check_deps.sh — verify python3.

Quick start

# Simple GET
python3 scripts/get.py https://api.example.com/users

# GET with auth, query params, JSON pretty-print
python3 scripts/get.py https://api.example.com/users \
    --bearer "$TOKEN" \
    --query 'limit=20' --query 'page=1' \
    --json

# Just the status code (great for health checks)
python3 scripts/get.py https://example.com --status-only

# POST JSON
python3 scripts/post.py https://api.example.com/users \
    --json '{"name":"Alice","email":"[email protected]"}' \
    --bearer "$TOKEN"

# POST a form
python3 scripts/post.py https://example.com/login \
    --form 'user=alice' --form 'pass=secret'

# PUT a JSON file
python3 scripts/post.py https://api.example.com/items/42 \
    --method PUT --json-file payload.json

# Download with progress + SHA-256 verification
python3 scripts/download.py \
    https://releases.example.com/model.tar.gz \
    ~/Downloads/model.tar.gz \
    --sha256 abc123def456...

# Resume an incomplete download
python3 scripts/download.py URL out.bin --resume

# Local static server for tests
python3 scripts/serve.py --directory ./public --port 8080

# Echo server for webhook smoke tests (auto-exits after 5 requests)
python3 scripts/serve.py --mode echo --port 9000 --max-requests 5

Exit codes

Safety properties

• Pure Python 3 standard library (urllib, http.server). No requests, no httpx, no pip install.
• All file paths validated against the same safe-path policy as the other clean-* toolkits.
• All URLs validated: must be http:// or https:// with a host.
• serve.py binds to 127.0.0.1 by default; you must pass --bind 0.0.0.0 explicitly to expose it.
• --insecure is opt-in only — TLS verification is on by default.
• Retries are bounded (--retries N, default 3) with exponential backoff capped at the backoff base × 2^attempt.

Why pure stdlib

Agents are usually run in restricted environments (sandboxes, CI runners, ephemeral containers) where pip install either fails, is forbidden, or burns minutes of setup time. Every script here uses only Python stdlib so it can run anywhere Python 3 is present.

Pairs well with

• clean-text-toolkit — pipe get.py URL into htmlstrip.py to scrape a page to plain text in one command.
• clean-json-toolkit — pipe API responses into query.py / validate.py / merge.py.
• clean-log-toolkit — serve.py static mode is useful for replaying captured log files over HTTP for parser tests.
• clean-csv-toolkit — download a CSV with download.py, then inspect.py / validate.py it.

v0.1.0 changes

• First public release. Four scripts: get.py, post.py, download.py, serve.py.
• Shared _common.py with safe_path, safe_url, parse_headers, fetch (with retry + gzip + redirect handling).
• All five clean-* toolkits now share the same safe-path policy, the same 0 / 1 / 2 exit-code contract, and the same zero-dependency principle.

License

MIT