ESG Reasoning Research Assistant

AI assistant for working on the ESG reasoning and green finance research project. This skill helps with experiment setup, evaluation, result interpretation, and documentation for DSPy prompt optimization research on the MMESGBench dataset.

Project Context

**Research Question**: Can DSPy prompt optimization match or exceed traditional fine-tuning (LoRA + RL) on ESG question answering with lower compute and fewer labels?

**Dataset**: MMESGBench - 933 ESG question-answer pairs from 45 corporate ESG reports

**Architecture**: PostgreSQL + pgvector retrieval → DSPy ChainOfThought reasoning → Structured answer extraction

**Key Models**: qwen-max, qwen2.5-7b-instruct with text-embedding-v4

Instructions

1. Initial Project Orientation

When starting work on this project:

1. **Read core documentation in order**:

- README.md - Current status and quick overview

- RESEARCH_FINDINGS.md - Complete analysis and recommendations

- CHANGELOG.md - Historical log of all work

- DEV_SET_ERROR_ANALYSIS.md - Error patterns by format type

2. **Verify authoritative data sources**:

- Data splits: Use ONLY `dspy_implementation/data_splits/` (train_186.json, dev_93.json, test_654.json)

- Results: Use files dated `20251019_130401` as authoritative

- Ignore old result files in other directories

3. **Check environment setup**:

- Conda environment: `esg_reasoning`

- Working directory: `/Users/victoryim/Local_Git/CC`

- Required env vars: `DASHSCOPE_API_KEY`, `PG_URL`, `ESG_COLLECTION_NAME`

2. Understanding Current Results

**Critical: Use Corrected Evaluator** (fixed Nov 9, 2025)

Always use the corrected evaluation function that recognizes null-equivalent responses:

```python

from src.evaluation import eval_score

answer_score = eval_score(gt, pred, answer_type)

correct = (answer_score >= 0.5) # ANLS 0.5 threshold

```

**Authoritative Dev Set Results (93 questions)**:

**Test Set Results (654 questions)**:

**Key Finding**: GEPA (reflection-based) outperforms MIPROv2 (teacher-student) on dev set. Reflection learns from actual student failures; teacher-student uses generic prompts that don't transfer well to 7B models.

3. Running Experiments

**Baseline Evaluation**:

```bash

python dspy_implementation/evaluate_baseline.py \

--model qwen2.5-7b-instruct \

--dataset dev \

--output baseline_dev_predictions.json

```

**GEPA Optimization**:

```bash

python dspy_implementation/gepa_skip_baseline.py

```

**Dynamic Cheatsheet Evaluation**:

```bash

Cold start (fair comparison to DSPy)

python dspy_implementation/dc_module/dc_evaluator.py \

--dataset test --variant cumulative

Warm start (test-time learning)

python dspy_implementation/dc_module/dc_evaluator.py \

--dataset test --variant cumulative --warmup

```

4. Code Quality Standards

For evaluation/optimization scripts (>10 min runtime), implement:

Reference: `archive_old_project/code_old/colbert_full_dataset_evaluation.py`

5. Common Pitfalls to Avoid

**DC vs DSPy Comparison**:

**Dataset Confusion**:

**Model Switching**:

**Old Result Files**:

**Optimizer Parameters**:

6. Key Research Insights

**Format-Specific Performance**:

**Prompt Length Trade-offs**:

**Cost-Performance**:

7. Documentation Updates

After completing work:

1. **Update README.md**: Quick status and new results

2. **Update RESEARCH_FINDINGS.md**: Detailed analysis and insights

3. **Update CHANGELOG.md**: Historical log entry

4. **Commit with clear message**: Include date and summary

**Don't create new documentation files** - update the existing three.

8. Pre-Flight Checklist

Before running experiments:

Before claiming results:

Database Configuration

**PostgreSQL with pgvector**:

Important Notes

1. **Evaluation Bug Fixes**: Two critical bugs were fixed (Nov 7 & Nov 9, 2025):

- Null equivalence: "null" now treated as equivalent to "Not answerable"

- ANLS string comparison: Fixed to use full string comparison

- Impact: +13.6% accuracy improvement for DC on test set

2. **Fair Comparisons**: Always compare like-to-like:

- Same dataset (dev/test/full)

- Same model (qwen-max/qwen2.5-7b)

- Same learning paradigm (cold start vs warm start)

3. **Research Contribution**: This work demonstrates that reflection-based optimization (GEPA) outperforms teacher-student (MIPROv2) for small language models (7B parameters), with significant cost-performance trade-offs.

Quick Links