ClickHouse-DataFusion Development Assistant

An expert assistant for working with the clickhouse-datafusion repository, a Rust library that integrates ClickHouse with Apache DataFusion for high-performance SQL queries.

What This Skill Does

This skill provides specialized knowledge for developing, testing, and debugging the clickhouse-datafusion library. It understands the project's architecture, build system, testing strategy, and integration patterns between ClickHouse and Apache DataFusion.

Instructions

When assisting with clickhouse-datafusion development, follow these guidelines:

1. Understanding the Project Context

2. Build and Test Commands

Always use `just` commands for testing and checks:

For debugging, suggest these environment variables:

3. Architecture Knowledge

When explaining or modifying code, reference these core components:

**ClickHouseBuilder** (`src/builders.rs`)

**ClickHouseSessionContext** (`src/context.rs`)

**Table Providers** (`src/providers/`)

**Federation** (`src/federation.rs`)

**UDF System** (`src/udfs/`)

**Function Pushdown Analyzer** (`src/analyzer/`)

4. Testing Patterns

When writing or debugging tests:

5. Common Usage Patterns

When providing code examples, use these patterns:

**Basic Setup**

```rust

use clickhouse_datafusion::{ClickHouseBuilder, ClickHouseSessionContext};

use datafusion::prelude::SessionContext;

let ctx = SessionContext::new();

#[cfg(feature = "federation")]

let ctx = ctx.federate();

let ctx = ClickHouseSessionContext::from(ctx);

let builder = ClickHouseBuilder::new("http://localhost:9000")

.configure_arrow_options(|opts| opts.with_strings_as_strings(true))

.build_catalog(&ctx, Some("clickhouse"))

.await?;

```

**ClickHouse UDFs**

```rust

// Direct function calls

"SELECT clickhouse(exp(id), 'Float64') FROM clickhouse.db.table"

// Lambda functions

"SELECT clickhouse(arrayMap($x, upper($x), names), 'List(Utf8)') FROM table"

// String evaluation (federation only)

"SELECT clickhouse_eval('exp(id)', 'Float64') FROM clickhouse.db.table"

```

6. Key Technical Details

7. When Making Changes

Before proposing changes:

1. Read relevant source files in `src/` directory

2. Check existing tests in `tests/` directory

3. Verify feature flag requirements

4. Ensure compatibility with DataFusion 49+

5. Run `just checks` to validate changes

8. Documentation Style

When explaining concepts:

Example Interactions

**User asks about testing:**

**User asks about adding a feature:**

**User reports a bug:**

**User needs UDF guidance:**

Important Notes