Ollamascope Telemetry Analysis

A bash-based telemetry wrapper for `ollama run` that captures per-run metrics including LLM performance, GPU utilization, and system resources. The entire project is a single self-contained bash script with no build process.

What This Skill Does

This skill helps you work with the Ollamascope codebase - a 210-line bash script that orchestrates metric collection around ollama runs. It captures GPU metrics (nvidia-smi), system metrics (dstat), and parses ollama's verbose output to extract token counts and performance data.

Instructions for AI Agent

When working with this codebase, follow these guidelines:

1. Understanding the Architecture

2. Core Workflow

When modifying or debugging the script:

1. **Footer Parsing (lines 27-44)**: Critical section that extracts metrics from ollama's verbose output using awk. Handles unit stripping and spacing variations.

2. **Process Management**: Background samplers (nvidia-smi, dstat) must be properly started before and killed after ollama execution

3. **Run ID Generation**: Format is ISO timestamp + hostname + model name + optional tag

4. **Error Handling**: Uses `set -euo pipefail` and trap for cleanup - maintain this pattern

3. Command Reference

```bash

Basic run with inline prompt

./ollamascope.sh --model <model:tag> --prompt "Your prompt here"

Run with prompt from file

./ollamascope.sh --model <model:tag> --prompt-file path/to/prompt.txt

Custom sampling interval (default 1s)

./ollamascope.sh --model <model:tag> --prompt "..." --interval 0.5

CPU-only systems (no GPU metrics)

./ollamascope.sh --model <model:tag> --prompt "..." --no-gpu

Tag runs for comparison

./ollamascope.sh --model <model:tag> --prompt "..." --tag warm-cache

```

4. Output Artifacts

Each run creates `runs/<run_id>/` containing:

5. Development Constraints

6. Dependencies

**Required:**

**Optional (auto-detected):**

7. Common Tasks

**Adding New Metrics:**

1. Identify source (ollama footer, nvidia-smi, dstat)

2. Update appropriate parsing section (footer: lines 27-44)

3. Add field to summary.json generation

4. Update README documentation

**Debugging Runs:**

1. Check `run.log` for raw ollama output

2. Verify footer format matches parsing regex

3. Ensure background processes started/stopped cleanly

4. Check CSV files for sampling continuity

**Modifying Sampling:**

1. Locate background process invocation

2. Adjust nvidia-smi or dstat command flags

3. Update CSV parsing if output format changes

4. Test with `--interval` flag variations

8. Important Notes

Usage Examples

**Example 1: Analyzing a specific run**

```

Read the summary.json from runs/<run_id>/ to understand performance metrics

```

**Example 2: Adding RAM usage to summary**

```

Parse sys_metrics.csv to calculate average RAM usage and add to summary.json

```

**Example 3: Debugging footer parsing**

```

Check lines 27-44 in ollamascope.sh and compare against run.log footer format

```