Remotion TikTok Captions

Build TikTok-style captioned videos with AI-powered transcription using Remotion, Whisper.cpp, and animated text overlays.

What This Skill Does

This skill helps you work with a Remotion-based video generation project that creates TikTok-style captioned videos with animated text overlays. The project uses Whisper.cpp for AI-powered speech-to-text transcription to automatically generate captions from video files with token-level precision and word-by-word highlighting.

Project Architecture

Core Pipeline

1. **Caption Generation (`sub.mjs`)**: AI transcription pipeline

- Extracts audio from video files using FFmpeg

- Transcribes audio using Whisper.cpp with token-level timestamps

- Generates caption JSON files stored in `public/` directory

- Supports batch processing of directories or individual files

2. **Video Composition (`src/CaptionedVideo/`)**: Remotion rendering components

- `index.tsx`: Main orchestrator for video playback and caption sequencing

- `SubtitlePage.tsx`: Caption entrance animations using spring physics

- `Page.tsx`: TikTok-style word-by-word color highlighting

- `NoCaptionFile.tsx`: Fallback UI when caption JSON is missing

3. **Caption System**: Page-based captions powered by `@remotion/captions`

- Captions split into pages (1200ms display time by default via `SWITCH_CAPTIONS_EVERY_MS`)

- Individual tokens highlighted as spoken

- Font pre-loaded using `load-font.ts` (TheBoldFont from `public/theboldfont.ttf`)

File Naming Convention

Paired files in `public/` directory:

Development Workflow

Start Development Server

```bash

npm run dev

```

Opens Remotion Studio with hot reload for live preview.

Generate Captions

**All videos in `public/`:**

```bash

npm run create-subtitles

```

**Specific video or folder:**

```bash

node sub.mjs <path-to-video-file>

node sub.mjs <path-to-folder>

```

Caption files are auto-generated and watched for changes during development.

Render Final Video

```bash

npx remotion render

```

Lint and Type-Check

```bash

npm run lint

```

Upgrade Remotion

```bash

npm run upgrade

```

Configuration

Whisper Settings (`whisper-config.mjs`)

**For non-English languages:** Remove `.en` suffix from model name and update `WHISPER_LANG`.

Caption Styling (`src/CaptionedVideo/Page.tsx`)

Remotion Settings (`remotion.config.ts`)

Key Files

| File | Purpose |

|------|---------|

| `sub.mjs` | Caption generation script (Whisper.cpp wrapper) |

| `whisper-config.mjs` | Whisper.cpp configuration |

| `remotion.config.ts` | Remotion video settings |

| `src/Root.tsx` | Available compositions (entry points) |

| `src/CaptionedVideo/index.tsx` | Main video component |

| `src/CaptionedVideo/Page.tsx` | Caption styling and highlighting |

| `src/CaptionedVideo/SubtitlePage.tsx` | Caption animations |

| `public/theboldfont.ttf` | Font file for captions |

Stack

First Run Setup

Common Tasks

Add New Video

1. Place video file in `public/` directory (e.g., `my-video.mp4`)

2. Generate captions: `node sub.mjs public/my-video.mp4`

3. Preview in Remotion Studio: `npm run dev`

4. Render: `npx remotion render`

Customize Caption Style

Edit `src/CaptionedVideo/Page.tsx`:

Change Caption Timing

Edit `SWITCH_CAPTIONS_EVERY_MS` in `src/CaptionedVideo/index.tsx` (default: 1200ms).

Use Different Whisper Model

Edit `whisper-config.mjs`:

Constraints

Example Use Cases

1. **Create social media content**: Generate TikTok/Instagram Reels-style captioned videos

2. **Accessibility**: Add auto-generated captions to existing video content

3. **Batch processing**: Generate captions for entire video libraries

4. **Multilingual content**: Transcribe and caption videos in multiple languages (change Whisper config)