GitHub Copilot Work Playground

Testing environment for personal/work usage with notebook-centric workflows for LLM-powered metadata generation.

Project Overview

This workspace specializes in generating and managing metadata using LLMs (OpenAI, Google Gemini) and MongoDB. All main logic is implemented in Jupyter notebooks, with reusable functions for data loading, sampling, LLM prompt construction, invocation, and MongoDB upload.

Instructions for AI Coding Agents

1. Data Loading and Sampling

**Always follow these patterns for data handling:**

Use `pandas.read_csv` to load CSV files

Sample up to 20 rows for LLM input (use all rows if fewer than 20)

Convert samples to JSON using `df.to_json(orient='records', force_ascii=False)`

If a file is empty, return an empty string and print a warning

```python

import pandas as pd

df = pd.read_csv('data.csv')

sample = df.head(20) if len(df) > 20 else df

json_data = sample.to_json(orient='records', force_ascii=False)

```

2. LLM Integration

**Support both OpenAI and Google Gemini:**

**OpenAI**: Use `langchain_openai.ChatOpenAI` with these defaults:

- `model="gpt-4o"`

- `temperature=0.2`

- `max_tokens=2000`

**Google Gemini**: Use `google-genai` with the latest model (e.g., `gemini-2.0-flash`)

Use system prompt + human prompt pattern (see `metadata_generator_system_prompt` and `metadata_info_prompt` in `MetadataAgent.ipynb`)

Extract JSON from LLM output using regex:

```python

import re

match = re.search(r'\{.*\}', llm_response, re.DOTALL)

if match:

json_str = match.group(0)

```

If LLM output is not valid JSON, print the raw response and error for debugging

3. Error Handling

**Handle failures gracefully:**

Catch and report API errors: quota limits, rate limits, token limits, credential errors

Print clear error messages for LLM and MongoDB operations

If LLM response cannot be parsed as JSON, print extracted string and parsing error

If API keys are missing, print warning and do not proceed

```python

try:

# LLM or MongoDB operation

except Exception as e:

print(f"Error: {e}")

```

4. Secrets and API Keys

**Never hardcode secrets:**

Always load from `.env` using `dotenv.load_dotenv()`

Required environment variables:

- `OPENAI_API_KEY`

- `GEMINI_API_KEY`

- `MONGODB_URI`

```python

from dotenv import load_dotenv

import os

load_dotenv()

openai_key = os.getenv('OPENAI_API_KEY')

gemini_key = os.getenv('GEMINI_API_KEY')

mongodb_uri = os.getenv('MONGODB_URI')

```

5. MongoDB Integration

**Database storage pattern:**

Use `pymongo` to connect with `MONGODB_URI`

Target database: `airspace`

Target collection: `metadata_full`

Use `insert_one()` to upload metadata JSON

Print inserted document ID on success

```python

from pymongo import MongoClient

client = MongoClient(os.getenv('MONGODB_URI'))

db = client['airspace']

collection = db['metadata_full']

result = collection.insert_one(metadata_json)

print(f"Inserted document ID: {result.inserted_id}")

```

6. Notebook Workflow

**Organize code in notebook cells:**

Implement all main logic in Jupyter notebook cells

Create reusable, well-documented functions for:

- Data loading

- Metadata generation

- Database upload

Use markdown cells to explain workflow and document assumptions

Keep functions modular and testable

7. Additional Info Handling

**Incorporate extra context when provided:**

If additional info (e.g., from `.txt` files) is available, append to LLM prompt

Use `add_info_prompt` template for incorporating extra context

Document the source and purpose of additional info

8. General Best Practices

**Follow these project conventions:**

Keep LLM input size small by sampling data (20 rows max)

Use `snake_case` for all identifiers in generated metadata

LLM output should contain **only** the final JSON (no extra text or explanations)

Document any new functions or workflows in markdown cells

Update this file if project patterns change

Example Workflow

1. Load environment variables from `.env`

2. Load CSV data with pandas

3. Sample up to 20 rows

4. Convert sample to JSON format

5. Construct LLM prompt with system + human messages

6. Invoke LLM (OpenAI or Gemini)

7. Extract and validate JSON from response

8. Upload metadata to MongoDB

9. Print confirmation with document ID

Key Files Reference

`MetadataAgent.ipynb` — Main notebook with metadata generation logic

`.env` — API keys and MongoDB URI (never commit!)

`metadata_generator_system_prompt` — System prompt template

`metadata_info_prompt` — Human prompt template

`add_info_prompt` — Template for additional context

Constraints

Always sample data to avoid token limits

Handle API errors gracefully with clear messaging

Never expose secrets in code or notebooks

Validate LLM output before database upload

Use markdown cells for documentation and workflow explanations

GitHub Copilot Work Playground

GitHub Copilot Work Playground

Project Overview

Instructions for AI Coding Agents

1. Data Loading and Sampling

2. LLM Integration

3. Error Handling

4. Secrets and API Keys

5. MongoDB Integration

6. Notebook Workflow

7. Additional Info Handling

8. General Best Practices

Example Workflow

Key Files Reference

Constraints

Reviews (0)