Nibi Cluster Archive Pipeline

A comprehensive guide for navigating and managing the file-based OCR pipeline on the Nibi HPC cluster. This skill helps you monitor, troubleshoot, and operate the Archive.org document processing pipeline with OLMoCR batch jobs.

What This Skill Does

This skill provides operational guidance for:

Pipeline Architecture

The system uses a **file-based state machine** (no SQLite) with 5 stages:

1. **01_downloaded/** - PDFs + metadata from Archive.org

2. **02_ocr_pending/** - Symlinks ready for OCR

3. **03_ocr_processing/** - Active batch directories

4. **04_ocr_completed/** - Split OCR results (per-page JSON)

5. **05_processed/** - Final processed documents

Three concurrent processes: Downloader → Dispatcher → Cleanup Worker

Instructions

1. Initial Connection

```bash

Connect from WSL

ssh nibi

If SSH hangs, check for stale multiplexer

ps aux | grep ssh

kill [PID]

```

2. Check Pipeline Status

```bash

BASE="/home/jic823/projects/def-jic823/caribbean_pipeline"

Quick counts

echo "Downloaded: $(ls $BASE/01_downloaded/*.pdf 2>/dev/null | wc -l)"

echo "Pending OCR: $(ls $BASE/02_ocr_pending/*.pdf 2>/dev/null | wc -l)"

echo "Processing: $(ls -d $BASE/03_ocr_processing/batch_* 2>/dev/null | wc -l)"

echo "Completed: $(ls $BASE/04_ocr_completed/*.jsonl 2>/dev/null | wc -l)"

echo "Final: $(ls $BASE/05_processed/*.json 2>/dev/null | wc -l)"

```

3. Monitor Active Jobs

```bash

List running jobs

squeue -u jic823

Detailed job info

squeue -u jic823 -l

Find and tail SLURM output

ls -lth ~/projects/def-jic823/slurm-*.out | head -5

tail -f ~/projects/def-jic823/slurm-[JOBID].out

```

4. Check Progress Manifests

```bash

Download progress

cat $BASE/_manifests/download_progress.json | jq '.'

OCR batch status

cat $BASE/_manifests/batches.json | jq '.batches[] | {batch_id, total_pdfs, status}'

Error count

find $BASE/99_errors -name "*.error.json" 2>/dev/null | wc -l

```

5. Verify OLMoCR Jobs

**CRITICAL:** Before submitting any OLMoCR job, read:

```

docs/OLMOCR_BEST_PRACTICES.md

```

This prevents silent failures where jobs process 1 PDF instead of hundreds.

```bash

Check OCR jobs

squeue -u jic823 | grep olmocr

Verify batch output

cat $BASE/03_ocr_processing/batch_0001/logs/*.log

```

6. Restart Pipeline After Failure

```bash

cd ~/projects/def-jic823/archive-olm-pipeline

git pull

Fully resumable - picks up from _manifests/download_progress.json

CONFIG_FILE=config/caribbean_filebased.yaml sbatch streaming/run_filebased_pipeline.sh

```

7. Troubleshooting

**Dispatcher not creating batches:**

```bash

Check pending count (triggers at 200 PDFs by default)

ls $BASE/02_ocr_pending/*.pdf | wc -l

Look for dispatcher log entries

grep "Dispatcher" ~/projects/def-jic823/slurm-[JOBID].out | tail -20

```

**Download stalled:**

```bash

Check progress

cat $BASE/_manifests/download_progress.json | jq '.current_index, .total_downloaded'

Check disk space (pauses at >90%)

df -h /home/jic823/projects/def-jic823

```

**Broken symlinks:**

```bash

find $BASE/02_ocr_pending -xtype l -delete

```

8. Health Indicators

**Healthy pipeline shows:**

**Warning signs:**

9. Emergency Commands

```bash

Cancel all jobs

scancel -u jic823

Kill hung SSH multiplexer

pkill -f "ssh.*nibi.*ControlMaster"

Nuclear reset (DELETES ALL PROGRESS)

rm -rf ~/projects/def-jic823/caribbean_pipeline

./setup_filebased_pipeline.sh

```

Key Configuration

**Pipeline config:** `config/caribbean_filebased.yaml`

Repository Locations

Performance Expectations

Constraints

Important Notes

1. **Always read** `docs/OLMOCR_BEST_PRACTICES.md` before submitting OCR jobs

2. Pipeline is fully resumable - safe to restart at any time

3. All state tracked in `_manifests/` JSON files, not SQLite

4. Dispatcher triggers based on PDF count (200), not page count

5. Cleanup worker auto-deletes PDFs to save space if enabled