Data Designer Agent

Expert agent for the Data Designer project: a cross-platform Rust application with gRPC microservices architecture for designing, testing, and managing dynamic data transformation rules using a soft DSL system for financial services.

What This Agent Does

This agent has deep knowledge of the Data Designer codebase, including:

Instructions

When working with this codebase, follow these guidelines:

Architecture Understanding

1. **Platform Layers:**

- Desktop UI: Native Rust with egui + wgpu + Tokio async runtime (`web-ui/src/main.rs`)

- Web UI: Pure Rust WASM client with egui (`web-ui/src/lib.rs`)

- Unified HTTP Client: Both platforms use identical reqwest-based HTTP layer (`web-ui/src/grpc_client.rs`)

- ~99.5% code sharing between desktop and WASM platforms

2. **Backend Services:**

- gRPC Server: Port 50051 (gRPC) + Port 8080 (HTTP REST fallback) in `grpc-server/`

- Onboarding Library: Metadata-driven workflow compiler and executor in `onboarding/`

- Core Library: Expression engine with database layer in `data-designer-core/`

3. **Database:**

- PostgreSQL database: `data_designer`

- Connection: `postgresql:///data_designer?user=adamtc007`

- 21 tables + 3 database views (cbu, cbu_investment_mandate_structure, cbu_member_investment_roles)

- Migrations in `database/migrations/`

- 100 entities + 8 CBUs in current dataset

Development Workflow

1. **Quick Start Commands:**

```bash

./rundesk.sh # Desktop application (best for development/debugging)

./runwasm.sh # Web application (WASM + browser)

cargo build # Build entire workspace

cargo test --all # Run comprehensive test suite

cargo clippy -- -D warnings # Enforce code quality

```

2. **Key Files to Check:**

- Frontend: `web-ui/src/cbu_dsl_ide.rs` (shared CBU DSL IDE)

- Backend: `grpc-server/src/main.rs` (gRPC + HTTP server)

- Workflow: `onboarding/src/planner/compile.rs` (compiler), `onboarding/src/runtime/scheduler.rs` (executor)

- Config: `onboarding/meta/*.yaml` (YAML-based configuration)

- Proto: `proto/financial_taxonomy.proto` (API definitions)

3. **Testing:**

- 20+ comprehensive tests including gRPC integration

- Parser, database, and gRPC integration test coverage

- Cargo clippy with `-D warnings` enforced

- Run full suite before major changes: `cargo test --all`

Code Quality Standards

1. **Clippy Compliance:**

- All code must pass `cargo clippy -- -D warnings`

- Use serde rename attributes for database field mapping

- Minimal unsafe code

- Comprehensive error handling with anyhow

2. **Cross-Platform Compatibility:**

- Use unified HTTP client pattern from `web-ui/src/grpc_client.rs`

- Minimize conditional compilation (#[cfg(target_arch = "wasm32")])

- Test both desktop and WASM builds

3. **Performance:**

- Maintain staged loading pattern (CBUs on startup, entities on-demand)

- Sub-second incremental builds expected

- 60fps UI performance target

Database Operations

1. **Schema Changes:**

- Create sequential migrations in `database/migrations/`

- Update schema exports in `database/schema/`

- Test with `cargo test --all` after migrations

2. **Entity Management:**

- Use database views for CBU/entity type mapping

- Leverage pgvector for semantic similarity queries

- Follow attribute-based dynamic schema pattern

Onboarding Library

1. **Workflow Compilation:**

- YAML configs in `onboarding/meta/` (CBU templates, product catalog, resource dictionaries)

- Sync compilation, async execution pattern

- IR-based plan generation with dependency resolution

2. **Task Orchestration:**

- Adapter stubs for gRPC, HTTP, and Kafka integration

- Error handling with detailed context

- Execution logging in `dsl_execution_logs` table

Common Tasks

1. **Adding New Features:**

- Create feature branch from `main`

- Update relevant workspace member (web-ui, grpc-server, onboarding, data-designer-core)

- Add tests in corresponding test modules

- Run `cargo clippy -- -D warnings` before commit

2. **Debugging:**

- Use desktop version for full IDE integration

- Check logs in gRPC server output

- Database queries via PostgreSQL client

- WASM debugging via browser console

3. **Building for Production:**

- Desktop: `cargo build --release --bin web-ui`

- WASM: `./web-ui/build-web.sh` (uses wasm-pack)

- Deploy WASM bundle from `web-ui/dist/`

Environment Configuration

Examples

Adding a New Database Table

1. Create migration: `database/migrations/00X_add_new_table.sql`

2. Update schema models in `data-designer-core/src/models/`

3. Add corresponding struct with serde attributes

4. Run migration and test: `cargo test --all`

Extending the DSL Parser

1. Update parser in `data-designer-core/src/parser/`

2. Add test cases in parser test module

3. Update CBU DSL IDE if UI changes needed

4. Test in both desktop and WASM builds

Adding a Workflow Step

1. Define in YAML config: `onboarding/meta/product_catalog.yaml`

2. Implement adapter in `onboarding/src/adapters/`

3. Add to task mapping in scheduler

4. Test via onboarding-cli: `cargo run --bin onboarding-cli`

Constraints