JavaScript ERM Prompt Optimization Framework

A JavaScript SDK implementation of the ERM (Exemplar and Reflection Memory) framework for optimizing Large Language Model prompts through iterative refinement.

Overview

This skill guides you in building and using a JavaScript-based prompt optimization framework that leverages feedback and exemplars to iteratively improve LLM prompt performance. The framework follows DRY and SOLID principles, emphasizing modularity, maintainability, and robust error handling.

Core Components

The framework consists of several key classes:

**`PromptBuilder`**: Incrementally constructs prompts with proper formatting

**`Feedback` and `Exemplar`**: Data classes representing feedback instances and examples with scoring

**`MemoryBase`**: Foundation class for memory management with score updating and Softmax probability calculations

**`FeedbackMemory` and `ExemplarFactory`**: Specialized memory managers for feedback and exemplars

**`ERM`**: Core optimizer orchestrating the evaluation and refinement loop

**`gen` Function**: Generic LLM API communication handler

Implementation Steps

1. Set Up Project Structure

Create a new JavaScript/Node.js project with the following structure:

```

project/

├── src/

│ ├── prompt_builder.js

│ ├── memory.js

│ ├── erm_optimizer.js

│ └── utils.js

├── config/

│ └── api_config.js

└── package.json

```

2. Implement Core Classes

**PromptBuilder**: Create a class that constructs prompts incrementally with methods like `addSection()`, `addExample()`, and `build()`.

**Memory Classes**: Implement `MemoryBase` with shared functionality for score management, then extend it with `FeedbackMemory` and `ExemplarFactory` for specialized storage and retrieval.

**ERM Optimizer**: Build the main optimization class that:

Accepts initial prompts and training/validation data

Runs evaluation loops

Collects feedback from failures

Generates exemplars from successes

Refines prompts iteratively

Returns optimized prompts with performance scores

3. Configure LLM Integration

Use dedicated SDKs (e.g., OpenAI SDK, Anthropic SDK)

Store API keys securely in environment variables

Implement asynchronous API calls with `async/await`

Enable streaming for large outputs when supported

Add rate limiting and throttling mechanisms

4. Follow Best Practices

**Code Quality**:

Apply DRY and SOLID principles

Use meaningful variable and function names

Add comments for complex logic

Follow ESLint style guide (Airbnb or Standard)

**Error Handling**:

Implement robust try-catch blocks

Validate and sanitize all inputs

Use structured logging (e.g., `winston`)

Avoid exposing sensitive information in errors

**Functional Programming**:

Favor immutability where appropriate

Use declarative programming patterns

Leverage array methods (`map`, `filter`, `reduce`)

5. Build Optimization Workflow

1. **Initialize**: Load initial prompt and data

2. **Evaluate**: Run prompt against validation data

3. **Collect Feedback**: Identify failures and generate feedback

4. **Store Exemplars**: Save successful examples with metadata

5. **Refine Prompt**: Use feedback and exemplars to improve prompt

6. **Iterate**: Repeat evaluation and refinement for N steps

7. **Return**: Output final prompt and performance metrics

6. Implement Helper Utilities

**Softmax**: Calculate probability distributions from scores

**Sample Indices**: Randomly select items based on probabilities

**Parsers**: Extract structured data from LLM responses (JSON, Markdown code blocks)

Example Usage

```javascript

const { ERM } = require('./erm_optimizer');

const logger = require('winston');

// Prepare data

const trainData = [

{ input: "Turn left at the next street.", target: "navigation" },

{ input: "E=mc^2", target: "physics" }

];

const valData = [

{ input: "The capital of France is Paris.", target: "geography" }

];

const testData = [

{ input: "Photosynthesis occurs in the chloroplast.", target: "biology" }

];

// Initial prompt

const initialPrompt = `

Task

Based on the input text, classify it into one of the predefined categories.

Output format

Provide the category label only.

Prediction

Text: {input}

Label:

`.trim();

// Optimize

const erm = new ERM();

const [finalScore, finalPrompt] = await erm.optimize(

initialPrompt,

trainData,

valData,

5 // optimization steps

);

logger.info(`Final validation score: ${finalScore}`);

logger.info(`Optimized prompt:\n${finalPrompt}`);

// Evaluate on test set

const testResults = await erm.evaluate(finalPrompt, testData);

const testScore = testResults.filter(r => r).length / testResults.length;

logger.info(`Test score: ${testScore}`);

```

Prompt Construction Guidelines

When building prompts for LLM optimization:

**Clear Format Instructions**: Specify exact output format (JSON, labels, etc.)

**Structured Sections**: Use Markdown headers (##) to organize prompt components

**Placeholder Variables**: Use `{variable}` syntax for dynamic content

**Exemplars**: Include successful examples in the prompt for few-shot learning

**Feedback**: Incorporate failure analysis to guide refinement

Configuration Parameters

**Temperature**: Control randomness in LLM responses (0.0 = deterministic, 1.0 = creative)

**Threshold**: Set minimum score for accepting improvements

**Beta**: Tune Softmax temperature for selection probability

**Max Feedback/Exemplars**: Limit memory size to prevent prompt bloat

Security and Performance

Store API keys in `.env` files or secure vaults

Never hardcode credentials

Implement rate limiting to respect API quotas

Use streaming for large text generation

Cache responses when appropriate

Log errors with structured logging libraries

Dependencies

Install required packages:

```bash

npm install winston dotenv

Add your LLM SDK (e.g., openai, @anthropic-ai/sdk)

```

Notes

The framework is designed for iterative prompt refinement, not one-shot generation

Performance depends on data quality and initial prompt design

Optimize on validation data, evaluate on held-out test data

Adjust hyperparameters (temperature, beta, threshold) based on your use case

The modular design allows easy extension for additional optimization strategies

JavaScript ERM Prompt Optimization