Zero-friction LLM telemetry capture tool - wraps ollama runs with GPU/CPU/RAM monitoring, parses metrics, and generates structured run artifacts for performance analysis
A bash-based telemetry wrapper for `ollama run` that captures per-run metrics including LLM performance, GPU utilization, and system resources. The entire project is a single self-contained bash script with no build process.
This skill helps you work with the Ollamascope codebase - a 210-line bash script that orchestrates metric collection around ollama runs. It captures GPU metrics (nvidia-smi), system metrics (dstat), and parses ollama's verbose output to extract token counts and performance data.
When working with this codebase, follow these guidelines:
When modifying or debugging the script:
1. **Footer Parsing (lines 27-44)**: Critical section that extracts metrics from ollama's verbose output using awk. Handles unit stripping and spacing variations.
2. **Process Management**: Background samplers (nvidia-smi, dstat) must be properly started before and killed after ollama execution
3. **Run ID Generation**: Format is ISO timestamp + hostname + model name + optional tag
4. **Error Handling**: Uses `set -euo pipefail` and trap for cleanup - maintain this pattern
```bash
./ollamascope.sh --model <model:tag> --prompt "Your prompt here"
./ollamascope.sh --model <model:tag> --prompt-file path/to/prompt.txt
./ollamascope.sh --model <model:tag> --prompt "..." --interval 0.5
./ollamascope.sh --model <model:tag> --prompt "..." --no-gpu
./ollamascope.sh --model <model:tag> --prompt "..." --tag warm-cache
```
Each run creates `runs/<run_id>/` containing:
**Required:**
**Optional (auto-detected):**
**Adding New Metrics:**
1. Identify source (ollama footer, nvidia-smi, dstat)
2. Update appropriate parsing section (footer: lines 27-44)
3. Add field to summary.json generation
4. Update README documentation
**Debugging Runs:**
1. Check `run.log` for raw ollama output
2. Verify footer format matches parsing regex
3. Ensure background processes started/stopped cleanly
4. Check CSV files for sampling continuity
**Modifying Sampling:**
1. Locate background process invocation
2. Adjust nvidia-smi or dstat command flags
3. Update CSV parsing if output format changes
4. Test with `--interval` flag variations
**Example 1: Analyzing a specific run**
```
Read the summary.json from runs/<run_id>/ to understand performance metrics
```
**Example 2: Adding RAM usage to summary**
```
Parse sys_metrics.csv to calculate average RAM usage and add to summary.json
```
**Example 3: Debugging footer parsing**
```
Check lines 27-44 in ollamascope.sh and compare against run.log footer format
```
Leave a review
No reviews yet. Be the first to review this skill!
# Download SKILL.md from killerskills.ai/api/skills/ollamascope-telemetry-analysis/raw