Lead Measure Analysis Guidance

AI guidance for developing the leadership measurement analysis project, which examines semantic distinctiveness of leadership constructs versus personality constructs using contrastive learning and embedding models.

Project Overview

This repository analyzes whether leadership measurement constructs are semantically distinct or redundant compared to established personality constructs (IPIP). The primary methodology uses sentence transformers with contrastive loss functions to evaluate construct separability.

Core Development Principles

Repository Management

1. **Avoid script proliferation**: Always search for and use existing scripts rather than creating new ones

2. **Extend, don't duplicate**: Enhance existing functionality instead of rewriting from scratch

3. **Maintain unified pipeline**: Use the consolidated `scripts/ivan_analysis/` approach for all new work

4. **Document thoughtfully**: Update existing documentation rather than creating separate files

5. **Clean as you go**: Use established cleanup procedures (`scripts/cleanup_models.py`, `scripts/cleanup_data.py`)

Code Development

1. **Check before creating**: Search the repository for existing implementations before writing new code

2. **Preserve functionality**: Ensure backward compatibility when modifying scripts

3. **Follow patterns**: Match existing code style, naming conventions, and project structure

4. **Test incrementally**: Validate changes against existing workflows

5. **Minimize complexity**: Choose simple solutions that integrate with current architecture

Key Directory Structure

```

.

├── analyses/ # Analysis scripts and code

├── data/

│ ├── raw/ # Original data files

│ ├── processed/ # Preprocessed datasets

│ └── visualizations/ # Output visualizations

├── docs/ # Documentation and reports

├── scripts/

│ └── ivan_analysis/ # Unified analysis pipeline (USE THIS)

├── models/ # Model checkpoints (~172GB, not in git)

└── manuscript/ # Research paper in Quarto format

```

Current Workflow Pipeline

The validated analysis pipeline consists of:

1. Generate Randomized Pairs

```bash

python3 scripts/ivan_analysis/generate_pairs_randomized.py

```

2. Create Holdout Splits

```bash

python3 scripts/ivan_analysis/create_holdout_splits.py --method stratified --holdout-ratio 0.1

```

- **Stratified item holdout** (recommended): 90-10 split across all constructs

- **Complete construct holdout**: Hold out ~10 entire constructs from training

3. Train Model with GIST Loss

```bash

python3 scripts/ivan_analysis/train_gist_unified.py --model-name bge-m3 \

--use-background --output-suffix final

```

4. Validate and Analyze

```bash

python3 scripts/ivan_analysis/validate_holdout.py

```

Validated Research Findings

IPIP Holdout Constructs (Never Seen During Training)

Leadership Constructs

Research Implications

Key Model Configurations

Optimized Training Parameters

Best Performing Models

Current Research Focus (June 2025)

PRIMARY: Manuscript Development

1. Update manuscript in `manuscript/leadership_measurement_paper.qmd`

2. Compile with Quarto: `quarto render manuscript/leadership_measurement_paper.qmd --to apaquarto-docx`

3. Incorporate key visualization: `top5_coherent_constructs_tsne.png`

4. Finalize statistical reporting with validated metrics (87.4% vs 62.9%)

SECONDARY: Analysis Refinement

1. Explore alternative taxonomies based on linguistic features

2. Apply dimensional reduction to identify parsimonious leadership dimensions

3. Create targeted visualizations for manuscript

FUTURE: Repository Organization

1. Create `/archive` folder for superseded analyses

2. Move visualizations to proper `/results` or `/outputs` folder

3. Consolidate scripts and reduce duplication

4. Clean up redundant visualization files

5. Streamline directory structure

Important Notes

Large Files Not in Git

Disk Space Management

When Working on This Project

1. **Before creating any new script**: Search `scripts/` for existing implementations

2. **Before starting analysis**: Check `scripts/ivan_analysis/` for the unified pipeline

3. **Before documenting**: Update existing docs rather than creating new files

4. **After completing work**: Run appropriate cleanup scripts if models/data were generated

5. **When in doubt**: Prefer the consolidated `ivan_analysis/` approach over scattered scripts

Research Recommendations

1. **Reconsider measurement approaches**: Acknowledge redundancy and develop more focused, distinctive measures

2. **Simplified framework**: Focus on 2-3 broader dimensions rather than 7-9 theoretically separate styles

3. **Different analytical lens**: View leadership styles as emphases on related traits, not discrete constructs