Granite 20B Function Calling (GGUF)

This skill provides access to IBM's Granite 20B Function Calling model in GGUF format, optimized for local deployment with llama.cpp and compatible inference engines. The model is specifically designed for function calling and tool use scenarios.

Model Overview

IBM Granite 20B Function Calling is a 20 billion parameter model trained for reliable function/tool calling. This quantized GGUF version enables running the model locally on consumer hardware while maintaining strong performance.

**Key Features:**

Quantization Options

The model is available in multiple quantization formats trading off size, speed, and quality:

Recommended Quants

Smaller Quants (Resource-Constrained)

Extreme Compression (Limited Use)

Usage Instructions

Step 1: Download the Model

Choose a quantization level based on your available VRAM/RAM and download from HuggingFace:

```bash

Example: Download Q4_K_M (recommended)

wget https://huggingface.co/mradermacher/granite-20b-functioncalling-i1-GGUF/resolve/main/granite-20b-functioncalling.i1-Q4_K_M.gguf

```

Step 2: Load with llama.cpp

```bash

Using llama-cli

./llama-cli -m granite-20b-functioncalling.i1-Q4_K_M.gguf \

--ctx-size 4096 \

--n-gpu-layers 35 \

--prompt "You are a helpful AI assistant with function calling capabilities."

```

Step 3: Function Calling Format

Structure your prompts to leverage the model's function calling abilities:

```

Available functions:

User: What's the weather in San Francisco?