LLM Connection & Tracing

Connect to LLM providers (OpenAI, Anthropic, etc.) and trace API calls. Includes vision utilities for querying images and detecting UI elements via bounding boxes.

What This Skill Does

This skill helps you integrate LLM APIs into your application with built-in tracing capabilities. It also provides vision utilities to:

Instructions

Basic LLM Connection

1. Install the package:

```bash

npm install @empiricalrun/llm

```

2. Import and initialize the LLM client:

```typescript

import { LLM } from "@empiricalrun/llm";

const llm = new LLM({

provider: "openai",

defaultModel: "gpt-4o",

});

```

3. Make a chat completion request:

```typescript

const llmResponse = await llm.createChatCompletion({

messages: [

{ role: "user", content: "Your prompt here" }

]

});

```

Vision Query (Extract Information from Images)

Use this to ask questions about an image and get text answers.

```typescript

import { query } from "@empiricalrun/llm/vision";

// Example with Appium screenshot (base64 encoded)

const data = await driver.saveScreenshot("dummy.png");

const instruction = "Extract number of ATOM tokens from the image. Return only the number.";

const text = await query(data.toString("base64"), instruction);

// Returns: "0.01"

```

**Use cases:**

Bounding Box Detection (Locate UI Elements)

Get coordinates for UI elements described in natural language.

```typescript

import { getBoundingBox } from "@empiricalrun/llm/vision/bbox";

const data = await driver.saveScreenshot("dummy.png");

const instruction = "This screenshot shows a screen to send crypto tokens. What is the bounding box for the dropdown to select the token?";

const bbox = await getBoundingBox(data.toString("base64"), instruction);

const centerToTap = bbox.center; // { x: 342, y: 450 }

```

**Important:** Coordinates are relative to the image dimensions. Scale them to your target system's coordinates before using (e.g., for Appium tap actions).

Debug Bounding Boxes

Iterate on bounding box detection with visual feedback:

```typescript

const bbox = await getBoundingBox(data.toString("base64"), instruction, {

debug: true,

});

// Returns base64 image with bounding box drawn

console.log(bbox.annotatedImage);

```

Save the annotated image to verify the detection is correct, then refine your instruction if needed.

Examples

Example 1: Test Automation with Vision

```typescript

// Verify a button exists and tap it

const screenshot = await driver.saveScreenshot("screen.png");

const bbox = await getBoundingBox(

screenshot.toString("base64"),

"Find the 'Continue' button"

);

await driver.touchAction({

action: "tap",

x: bbox.center.x * scaleX,

y: bbox.center.y * scaleY

});

```

Example 2: Extract Data from Screenshots

```typescript

const screenshot = await takeScreenshot();

const balance = await query(

screenshot,

"What is the account balance shown? Return only the number."

);

console.log(`Current balance: $${balance}`);

```

Notes