Polars DataFrame Analysis

Expert skill for working with Polars, the blazingly fast DataFrame library written in Rust. Use this for high-performance data manipulation, transformation, and analytical queries with datasets of any size.

What This Skill Does

Helps you leverage Polars' powerful query engine for DataFrame operations including:

Prerequisites

Install Polars:

```bash

pip install polars

```

For optional features:

```bash

All optional dependencies

pip install polars[all]

Streaming for larger-than-RAM datasets

pip install polars[streaming]

Timezone support

pip install polars[timezone]

Excel support

pip install polars[excel]

```

Step-by-Step Instructions

1. Import and Basic Setup

Start by importing Polars and checking the installation:

```python

import polars as pl

Check version and available features

pl.show_versions()

```

2. Reading Data

Read data from various sources:

```python

From CSV

df = pl.read_csv("data.csv")

From Parquet (recommended for performance)

df = pl.read_parquet("data.parquet")

From JSON

df = pl.read_json("data.json")

Lazy reading (for query optimization)

lazy_df = pl.scan_csv("large_data.csv")

lazy_df = pl.scan_parquet("large_data.parquet")

```

3. Basic DataFrame Operations

Perform common transformations:

```python

Select columns

df.select(["column1", "column2"])

Filter rows

df.filter(pl.col("age") > 30)

Add/modify columns

df.with_columns([

(pl.col("price") * 1.1).alias("price_with_tax"),

pl.col("name").str.to_uppercase().alias("name_upper")

])

Sort

df.sort("column_name", descending=True)

Group by and aggregate

df.group_by("category").agg([

pl.col("sales").sum().alias("total_sales"),

pl.col("quantity").mean().alias("avg_quantity")

])

```

4. Lazy Execution (Query Optimization)

Use lazy evaluation for automatic query optimization:

```python

Create lazy frame

lazy_df = pl.scan_csv("data.csv")

Chain operations (not executed yet)

result = (

lazy_df

.filter(pl.col("year") >= 2020)

.group_by("region")

.agg(pl.col("revenue").sum())

.sort("revenue", descending=True)

.head(10)

)

Execute the optimized query

final_df = result.collect()

For larger-than-RAM datasets, use streaming

final_df = result.collect(streaming=True)

```

5. Expressions API

Leverage Polars' powerful expression system:

```python

df.select([

# Conditional logic

pl.when(pl.col("age") < 18)

.then(pl.lit("minor"))

.otherwise(pl.lit("adult"))

.alias("age_group"),

# String operations

pl.col("email").str.extract(r"@(.+)$", 1).alias("domain"),

# Date operations

pl.col("date").dt.year().alias("year"),

pl.col("date").dt.month().alias("month"),

# Window functions

pl.col("sales").rank().over("region").alias("sales_rank"),

# Rolling operations

pl.col("value").rolling_mean(window_size=7).alias("7day_avg")

])

```

6. Joins and Concatenation

Combine DataFrames:

```python

Inner join

result = df1.join(df2, on="id", how="inner")

Left join with custom keys

result = df1.join(df2, left_on="user_id", right_on="id", how="left")

Vertical concatenation (stacking)

combined = pl.concat([df1, df2], how="vertical")

Horizontal concatenation

combined = pl.concat([df1, df2], how="horizontal")

```

7. Writing Data

Export results:

```python

To Parquet (recommended)

df.write_parquet("output.parquet")

To CSV

df.write_csv("output.csv")

To JSON

df.write_json("output.json")

To pandas (if needed)

pandas_df = df.to_pandas()

```

8. Performance Tips

Optimize your Polars queries:

```python

Use lazy execution for automatic optimization

lazy_df = pl.scan_parquet("data.parquet")

Use streaming for large datasets

result = lazy_df.collect(streaming=True)

Prefer Parquet over CSV for reading/writing

df.write_parquet("data.parquet", compression="zstd")

Use filter early in the pipeline

lazy_df.filter(pl.col("date") > "2023-01-01") # Filter before other ops

Use select to reduce memory footprint

lazy_df.select(["col1", "col2"]) # Only needed columns

```

Common Patterns

Data Cleaning

```python

df.select([

pl.col("*").fill_null(strategy="forward"), # Fill nulls

pl.col("numeric_col").clip(0, 100), # Clip values

pl.col("text_col").str.strip(), # Remove whitespace

])

```

Time Series Analysis

```python

df.sort("timestamp").select([

pl.col("timestamp"),

pl.col("value").diff().alias("change"),

pl.col("value").pct_change().alias("pct_change"),

pl.col("value").rolling_mean(window_size=7).alias("7day_ma")

])

```

Pivot and Unpivot

```python

Pivot

pivoted = df.pivot(values="sales", index="date", columns="product")

Unpivot/melt

melted = df.melt(id_vars=["id", "date"], value_vars=["sales", "profit"])

```

Key Advantages

When to Use Polars

✅ Large datasets (>1GB)

✅ Performance-critical pipelines

✅ Complex aggregations and joins

✅ Larger-than-RAM data processing

✅ New projects with no pandas dependency

⚠️ Consider pandas if you need extensive ecosystem compatibility or have legacy code

Resources