Guides work on a Next.js 15 web scraper using Firecrawl API to map, analyze, and migrate website content with intelligent schema inference and batch extraction.
A Next.js 15 web application for scraping and migrating content from websites using the Firecrawl API. Built with TypeScript, React 19, and shadcn/ui components.
You are working with a web scraping application that helps users:
1. Map website structure to discover URLs
2. Analyze and categorize content (posts, pages, categories, tags)
3. Infer schemas from sample pages
4. Batch scrape selected URLs with structured data extraction
5. Export extracted data as JSON or CSV
When working with this project:
1. **Install dependencies**: `npm install`
2. **Run development server**: `npm run dev`
3. **Build for production**: `npm run build`
4. **Start production server**: `npm start`
5. **Run linting**: `npm run lint`
Always ensure `.env.local` contains:
```env
FIRECRAWL_API_KEY=fc-YOUR_ACTUAL_API_KEY_HERE
```
Validate this key exists in all API route handlers.
- `/api/map/` - Website mapping endpoint
- `/api/crawl/` - Content scraping with schema inference
- `/ui/` - shadcn/ui components (40+ pre-built)
- `/layout/` - Layout components
**`/api/map/route.ts`**
**`/api/crawl/route.ts`**
1. **TypeScript**: Use strict mode, maintain type safety
2. **Imports**: Use `@/` alias for src directory imports
3. **Components**:
- Name files in kebab-case: `react-component-like-this.tsx`
- Use shadcn/ui components when available
4. **Styling**:
- Apply Tailwind CSS classes directly
- Use `cn()` helper from `/src/lib/utils.ts` for conditional classes
- Avoid custom CSS files
5. **API Routes**: Always validate `FIRECRAWL_API_KEY` from environment
6. **Error Handling**: Implement fallbacks for batch operations
1. **Read before modifying**: Always read existing files before making changes
2. **Maintain patterns**: Follow existing architectural patterns
3. **Check environment**: Verify API key configuration for API routes
4. **Test endpoints**: Validate both mapping and crawling workflows
5. **Component consistency**: Use established shadcn/ui components
6. **Type safety**: Ensure TypeScript strict mode compliance
No test framework is currently configured. Before implementing tests:
1. Check `package.json` for existing test dependencies
2. Review project structure for test conventions
3. Propose appropriate testing approach (Jest, Vitest, etc.)
Leave a review
No reviews yet. Be the first to review this skill!
# Download SKILL.md from killerskills.ai/api/skills/firecrawl-content-migration/raw