EverQuest Data Browser Dev Guide
Development guidance for working with EQDataScraper, a full-stack Vue.js application for browsing EverQuest spell data.
Project Architecture
**Tech Stack:**
Frontend: Vue 3 SPA with Vue Router, Pinia, Vite (port 3000 default)Backend: Flask REST API (port 5001 default)Data: Web scraping from alla.clumsysworld.comDatabases: PostgreSQL (auth/cache) + MySQL (game data, read-only)**Key Directories:**
`src/` - Vue frontend (App.vue, views/, stores/, router/)`backend/` - Flask API (app.py)`cache/` - Persistent JSON cache (gitignored)Development Workflow
1. **Start Development**
- Run `python3 run.py install` to install all dependencies
- Run `python3 run.py start` to start both frontend and backend
- For testing: Set `ENABLE_DEV_AUTH=true` and `ENABLE_USER_ACCOUNTS=true`
2. **Individual Services**
- Frontend only: `npm run dev` (Vite dev server)
- Backend only: `cd backend && python app.py`
- Check status: `python3 run.py status`
- Stop all: `python3 run.py stop`
3. **Building**
- Production build: `npm run build`
- Preview build: `npm run preview`
- Never commit `dist/` - Railway builds automatically
Important API Patterns
**CRITICAL - API URL Configuration:**
Always use `API_BASE_URL` in Vue components for production compatibility:
```javascript
const API_BASE_URL = import.meta.env.VITE_BACKEND_URL ||
(import.meta.env.PROD ? 'https://backend-url.railway.app' : '')
const response = await axios.get(`${API_BASE_URL}/api/endpoint`)
```
Never use relative `/api/` URLs - they only work in development with Vite proxy.
**Proper Response Handling:**
```javascript
// Always capture response data
const response = await axios.get(`${API_BASE_URL}/api/spells/${className}`)
const actualCount = response.data.spell_count || 0
// Detect HTML responses (routing issues)
if (typeof response.data === 'string' && response.data.includes('<!DOCTYPE html>')) {
throw new Error('API returned HTML - proxy routing issue')
}
```
Database Architecture (CRITICAL)
**Two Separate Systems - DO NOT CONFUSE:**
**1. Auth Database (PostgreSQL)**
Purpose: User auth, OAuth, activity logs, AND scraped spell cacheConnection: `DATABASE_URL` environment variableAccess: Read/WriteTables: `users`, `oauth_sessions`, `user_preferences`, `activity_logs`, spell cacheConfigure: Railway environment variables ONLY**2. Content Database (MySQL)**
Purpose: EverQuest game data (items, discovered_items)Connection: `production_database_url` in config.jsonAccess: READ-ONLY (enforced)Tables: `items`, `discovered_items` (EQEmu database)Configure: Admin Dashboard → Database Configuration ONLY**NEVER:**
Configure content database using `DATABASE_URL`Point auth database to EQEmu MySQLWrite to content databaseAPI Endpoints
`GET /api/classes` - List all EverQuest classes`GET /api/spells/{className}` - Get spells for specific class`GET /api/search-spells?q={query}` - Search across all classes`GET /api/spell-details/{spellId}` - Get detailed spell info`POST /api/scrape-all` - Trigger scraping for all classes`GET /api/cache-status` - Check cache status`GET /api/health` - Health check`GET /api/cache/status` - Cache statistics`POST /api/cache/save` - Manual cache save`POST /api/cache/clear` - Clear all cacheCaching System
**Persistent Cache:**
Location: `cache/` directory (gitignored)Files: `spells_cache.json`, `pricing_cache.json`, `cache_metadata.json`Benefits: Survives restarts, faster startup, no re-scraping on deployAuto-save: After scrapes and every 10 pricing updatesExpiry: 24 hours default**Cache Management:**
Check status via API or admin endpointsClear manually when data structure changesRate limited: 5-minute minimum between scrapesGit Workflow
**ALWAYS use feature branches - NEVER commit directly to master:**
```bash
Create branch
git checkout -b feature/spell-search
Push and create PR
git push -u origin feature/spell-search
gh pr create
Open PR in Chrome automatically
open -a "Google Chrome" <PR_URL>
```
**Branch Naming:**
`feature/description` - New functionality`fix/description` - Bug fixes`refactor/description` - Code improvements`docs/description` - Documentation**Commit Guidelines:**
All commits attributed to repository owner ONLYNEVER include Co-Authored-By tagsNEVER mention Claude, AI, or AI assistanceUse conventional format: `feat:`, `fix:`, `refactor:`Keep messages concise and descriptiveReusable Components
**LoadingModal (`src/components/LoadingModal.vue`):**
Use for all loading states (except class card internals):
```vue
<LoadingModal :visible="isLoading" />
<LoadingModal :visible="isSaving" text="Saving changes..." />
<LoadingModal :visible="isProcessing" :full-screen="true" />
```
Features:
Random EverQuest class icons with pulsing animationGlassmorphism theme matchingConfigurable text and positioningProps:
`visible` (Boolean) - Show/hide`text` (String) - Loading message`fullScreen` (Boolean) - Viewport coverage`showIcon` (Boolean) - Display icon`customIcon` (String) - Custom icon URL`randomClassIcon` (Boolean) - Random EQ iconsRailway Deployment Best Practices
**Common Issues:**
1. **API routing fails** - Symptoms: HTML responses, 404s, JSON parsing errors
- Fix: Always use `API_BASE_URL` in components
- Test: `curl` backend endpoints directly
2. **Stale frontend code** - Changes not appearing
- Fix: Force browser cache refresh (Ctrl+Shift+R)
- Check: Search minified JS for key strings
3. **Build artifacts in git** - `dist/` committed
- Fix: Add to `.gitignore`, clean git history
- Note: Railway builds automatically
**Testing Checklist:**
Direct backend API tests via curlBrowser console for routing errorsFrontend build includes expected changesCache refresh after bulk operationsConfiguration
**config.json:**
Port settings for frontend/backendCache expiry and scrape intervalsContent database connection (MySQL)**Environment Variables:**
`DATABASE_URL` - Auth database (PostgreSQL)`BACKEND_PORT` / `FRONTEND_PORT` - Port overrides`ENABLE_USER_ACCOUNTS` - Enable auth features`ENABLE_DEV_AUTH` - Bypass auth for testing`JWT_SECRET_KEY` - Auth token generation`GOOGLE_CLIENT_ID` / `GOOGLE_CLIENT_SECRET` - OAuth`CACHE_EXPIRY_HOURS` - Cache lifetime`MIN_SCRAPE_INTERVAL_MINUTES` - Rate limit**Port Conflicts:**
Automatically detected by `run.py`macOS: Port 5000 conflicts with AirPlay ReceiverCross-Platform Support
Windows: `run.bat`, `taskkill`, `netstat`, `tasklist`macOS: `lsof`, `ps`, POSIX signalsLinux: `ss`/`netstat`, POSIX signalsWSL: Automatically detectednpm: Platform-specific `.cmd` handlingData Flow
1. Vue frontend requests spell data via Axios
2. Flask checks persistent cache (24h expiry)
3. On cache miss: scrapes alla.clumsysworld.com with BeautifulSoup
4. Data cached in memory and saved to JSON
5. Frontend stores in Pinia with class theming
6. Scraped data also stored in PostgreSQL auth database
UI Design Patterns
Dynamic CSS classes for conditional styling (`has-pagination`)Fixed-height components with graceful degradationFlexbox layouts to prevent overflowMobile-responsive from the startGlassmorphism aesthetic throughoutGlobal Search Implementation
Cross-class deduplication by `spell_id`Consistent class abbreviation mappingPagination triggers at 10+ results (520px dropdown)Hash navigation with query parameters for deep linkingModal auto-opening with timing coordinationClass System
16 EverQuest classes with unique color themes stored in `stores/spells.js`. Class names normalized to lowercase for consistent caching and API calls.
Testing and Debugging
When testing features requiring auth or admin access:
```bash
export ENABLE_DEV_AUTH=true
export ENABLE_USER_ACCOUNTS=true
python3 run.py start
```
This enables:
Admin route accessSearch event loggingMock data and auth bypassAll debugging features**Debug Logging:**
Comprehensive console logs for data operationsLog both success and failure with data countsInclude error details and response types