Quantized GGUF version of DeepSeek Coder 6.7B fine-tuned for chat and function calling with OpenAI-compatible syntax. Multiple quantization levels available from 2.36GB to 6.67GB.
Quantized GGUF versions of DeepSeek Coder 6.7B fine-tuned for conversational interactions and function calling capabilities. This model uses OpenAI-compatible syntax and can intelligently return function calls when appropriate.
This model was created by fine-tuning deepseek-coder-6.7b on the Open Assistant dataset, followed by additional training on function calling data. It provides code generation, chat capabilities, and structured function calling in a compact 6.7B parameter model.
**Original Model:** [AIGym/deepseek-coder-6.7b-chat-and-function-calling](https://huggingface.co/AIGym/deepseek-coder-6.7b-chat-and-function-calling)
**Quantization by:** [RichardErkhov](https://github.com/RichardErkhov)
Choose the quantization level based on your hardware constraints and quality requirements:
| Quantization | Size | Use Case |
|-------------|------|----------|
| Q2_K | 2.36GB | Minimal memory, lowest quality |
| IQ3_XS | 2.61GB | Very constrained devices |
| Q3_K_M | 3.07GB | Balance for low-memory systems |
| Q4_K_M | 3.80GB | Recommended for most users |
| Q5_K_M | 4.46GB | Higher quality, moderate size |
| Q6_K | 5.15GB | Near-original quality |
| Q8_0 | 6.67GB | Highest quality quantized |
Download your preferred quantization from the [model repository](https://huggingface.co/RichardErkhov/AIGym_-_deepseek-coder-6.7b-chat-and-function-calling-gguf).
**Recommendation:** Start with Q4_K_M for the best balance of quality and size.
**llama.cpp:**
```bash
./main -m deepseek-coder-6.7b-chat-and-function-calling.Q4_K_M.gguf -p "Your prompt here" -n 512
```
**Ollama:**
```bash
echo 'FROM ./deepseek-coder-6.7b-chat-and-function-calling.Q4_K_M.gguf' > Modelfile
ollama create deepseek-coder-function:6.7b -f Modelfile
ollama run deepseek-coder-function:6.7b
```
**LM Studio / Jan / GPT4All:**
Import the GGUF file through the application's model import interface.
This model is trained to work with OpenAI-style function calling syntax. Structure your prompts to include function definitions when you want the model to return structured function calls.
**Example Prompt Structure:**
```
You are a helpful assistant with access to the following functions:
[Function definitions in JSON format]
User: [User query]
```
The model will return function calls in the appropriate format when it determines a function should be invoked.
When the model returns a function call:
1. Parse the function name and arguments from the model's response
2. Execute the function in your application
3. Return the result to the model as a function response
4. Continue the conversation
| Benchmark | Score |
|-----------|-------|
| AI2 Reasoning Challenge (25-shot) | 36.09 |
| HellaSwag (10-shot) | 53.80 |
| MMLU (5-shot) | 38.29 |
| TruthfulQA (0-shot) | 42.83 |
| Winogrande (5-shot) | 57.22 |
| GSM8k (5-shot) | 17.21 |
| **Average** | **40.91** |
Apache 2.0
Leave a review
No reviews yet. Be the first to review this skill!
# Download SKILL.md from killerskills.ai/api/skills/deepseek-coder-67b-chat-and-function-calling-gguf/raw