Firecrawl Content Migrator

A Next.js 15 web application for scraping and migrating content from websites using the Firecrawl API. Built with TypeScript, React 19, and shadcn/ui components.

Project Context

You are working with a web scraping application that helps users:

1. Map website structure to discover URLs

2. Analyze and categorize content (posts, pages, categories, tags)

3. Infer schemas from sample pages

4. Batch scrape selected URLs with structured data extraction

5. Export extracted data as JSON or CSV

Technology Stack

**Framework**: Next.js 15.3.5 with App Router

**Language**: TypeScript (strict mode)

**UI**: shadcn/ui components (Radix UI + Tailwind CSS)

**API**: Firecrawl SDK (@mendable/firecrawl-js)

**Styling**: Tailwind CSS with tailwind-merge and class-variance-authority

**Dev Server**: Turbopack

Development Commands

When working with this project:

1. **Install dependencies**: `npm install`

2. **Run development server**: `npm run dev`

3. **Build for production**: `npm run build`

4. **Start production server**: `npm start`

5. **Run linting**: `npm run lint`

Environment Configuration

Always ensure `.env.local` contains:

```env

FIRECRAWL_API_KEY=fc-YOUR_ACTUAL_API_KEY_HERE

```

Validate this key exists in all API route handlers.

Architecture Guidelines

Directory Structure

`/src/app/` - App Router pages and API routes

- `/api/map/` - Website mapping endpoint

- `/api/crawl/` - Content scraping with schema inference

`/src/components/` - React components

- `/ui/` - shadcn/ui components (40+ pre-built)

- `/layout/` - Layout components

`/src/lib/` - Utilities (cn helper for className merging)

`/src/hooks/` - Custom React hooks

API Routes

**`/api/map/route.ts`**

Maps website structure via Firecrawl's mapUrl

Categorizes URLs (posts, categories, tags, pages)

Returns analyzed URL list with metadata

**`/api/crawl/route.ts`**

**GET**: Infers schema from sample page content

**POST**: Batch scrapes URLs with schema extraction

Supports automatic schema inference (prices, dates, authors)

Falls back to markdown when structured extraction fails

Frontend Patterns

Multi-step workflow: map → select → configure schema → scrape → export

Client-side state for mapping, selection, schema, and results

Tree view for URL selection

Collapsible JSON viewer for data inspection

CSV export for structured data

Schema System

Automatic inference detects content patterns (prices, dates, authors, categories)

Supports: string, number, boolean, array types

Converts between internal format and Firecrawl JSON schema

Handles both structured extraction and markdown fallback

Code Style Requirements

1. **TypeScript**: Use strict mode, maintain type safety

2. **Imports**: Use `@/` alias for src directory imports

3. **Components**:

- Name files in kebab-case: `react-component-like-this.tsx`

- Use shadcn/ui components when available

4. **Styling**:

- Apply Tailwind CSS classes directly

- Use `cn()` helper from `/src/lib/utils.ts` for conditional classes

- Avoid custom CSS files

5. **API Routes**: Always validate `FIRECRAWL_API_KEY` from environment

6. **Error Handling**: Implement fallbacks for batch operations

When Making Changes

1. **Read before modifying**: Always read existing files before making changes

2. **Maintain patterns**: Follow existing architectural patterns

3. **Check environment**: Verify API key configuration for API routes

4. **Test endpoints**: Validate both mapping and crawling workflows

5. **Component consistency**: Use established shadcn/ui components

6. **Type safety**: Ensure TypeScript strict mode compliance

Testing

No test framework is currently configured. Before implementing tests:

1. Check `package.json` for existing test dependencies

2. Review project structure for test conventions

3. Propose appropriate testing approach (Jest, Vitest, etc.)

Important Notes

All API routes must validate Firecrawl API key from environment variables

Batch scraping implements fallback to individual requests on failure

Schema inference analyzes content for e-commerce, blog, and business patterns

Dual-format processing (markdown + HTML) ensures robust extraction

Follow Next.js 15 App Router conventions strictly

Firecrawl Content Migration