News Aggregator Code Assistant

Expert guidance for working with a modern AI-focused news aggregator built with Node.js/Express backend and vanilla JavaScript frontend. Specializes in dynamic source management, adaptive scraping, and content filtering.

Project Context

This is a comprehensive news aggregator (~1600 lines frontend) focusing on AI industry news, research, tools, and coding platforms. Features include:

Available Commands

Development & Production

Testing & Quality

Key Files & Architecture

Backend

Frontend

Testing

Step-by-Step Workflow

When Asked to Add/Modify Features

1. **Identify affected components**

- Backend: Check if route changes needed in `server.js`

- Scraping: Check if new scraper needed in `newsService.js`

- Frontend: Determine if UI changes needed in `app.js` or HTML

- Sources: Determine if `sources.json` needs updates

2. **Review current architecture**

- Read existing implementation in relevant files

- Check data flow: sources.json → newsService → server.js → API → frontend

- Review security patterns in similar existing code

3. **Implement changes**

- Apply consistent code style with existing patterns

- Add input sanitization for any user input

- Update API endpoints if needed

- Maintain class-based structure in frontend

4. **Update configuration**

- Add new sources to `sources.json` with proper category

- Include status tracking fields: `articleCount`, `lastSuccess`, `lastError`, `lastAttempt`

- Set `status: "active"` for enabled sources

5. **Test thoroughly**

- Run `npm test` to verify existing tests pass

- Add new tests if implementing security-sensitive features

- Test API endpoints manually with curl or browser

- Verify source management in admin interface

When Asked About News Sources

1. **Check `sources.json`** for current configuration

2. **Categories available**: AI Industry, AI News, AI Research, Coding Tools

3. **For new sources**, guide user to:

- Add entry to `sources.json` with unique ID

- Test with generic scraper first

- Implement custom scraper in `newsService.js` if needed

4. **Source fields required**:

```json

{

"id": "unique-id",

"name": "Display Name",

"url": "https://example.com",

"category": "AI Industry",

"status": "active"

}

```

When Asked About Security

1. **Review existing sanitization functions**:

- `escapeHtml()` - Prevents XSS in text content

- `sanitizeHtml()` - Removes dangerous HTML elements

- `sanitizeUrl()` - Validates HTTP/HTTPS URLs only

- `sanitizeSearchInput()` - Limits search input length and patterns

2. **For new inputs**, ensure:

- All user input passes through appropriate sanitizer

- API endpoints validate and sanitize before processing

- Prototype pollution protection for object updates

- Test coverage exists in `tests/sanitizer.test.js`

3. **Run security tests**: `npm run test:security`

When Asked About Testing

1. **Understand test architecture**:

- Custom lightweight test framework (no external dependencies)

- API tests require server running on port 3000

- Tests include automatic health check with retry logic

2. **Add tests for new features**:

- Security-sensitive code → `tests/sanitizer.test.js`

- API endpoints → `tests/api.test.js`

- Follow existing test patterns

3. **Test execution flow**:

```bash

npm start # Start server in separate terminal

npm test # Run all tests

npm run test:api # API tests only

```

When Debugging Issues

1. **Check logs and error handling**:

- Server logs in console output

- Source status in `sources.json` (`lastError`, `lastAttempt`)

- Browser console for frontend errors

2. **Common issues**:

- **No articles**: Check if sources are `"active"` in `sources.json`

- **Scraping failures**: Review source-specific scraper or add custom one

- **API errors**: Verify server running, check CORS configuration

- **Tests failing**: Ensure server running on port 3000 for API tests

3. **Debugging data flow**:

- Verify `sources.json` structure

- Check `newsService.js` scraper selection logic

- Inspect cache state in `server.js`

- Test API endpoints directly with curl

Important Notes

Examples

Adding a New News Source

```bash

1. Edit sources.json, add entry:

{

"id": "new-source",

"name": "New AI Blog",

"url": "https://newblog.ai",

"category": "AI Industry",

"status": "active",

"created": "2025-02-03"

}

2. Restart server to pick up changes

npm run dev

3. Test via admin interface at /admin

4. If generic scraper fails, implement custom scraper in newsService.js

```

Running Complete Test Suite

```bash

Terminal 1: Start server

npm start

Terminal 2: Run tests

npm test

Check exit code (0 = success, 1 = failure)

echo $?

```

Debugging Scraper Issues

```bash

1. Check server logs for scraping errors

npm run dev

2. Inspect sources.json for lastError field

3. Test individual source URL manually

4. Add custom scraper if needed in newsService.js

```

Key Constraints