MLX Llama-3 Tool-Use Generator

Generate text responses with Groq's Llama-3-Groq-8B-Tool-Use model, specifically fine-tuned for function calling and tool use scenarios. This skill uses the MLX-optimized 4-bit quantized version for efficient inference on Apple Silicon.

What This Skill Does

This skill loads and runs the `mlx-community/Llama-3-Groq-8B-Tool-Use-4bit` model using the MLX framework. The model is based on Meta's Llama-3 8B architecture and has been fine-tuned by Groq for:

The 4-bit quantization allows the 8B parameter model to run efficiently on Apple Silicon (M1/M2/M3) with reduced memory footprint while maintaining strong performance.

Instructions

When the user requests text generation with function calling capabilities or tool use:

1. **Install Dependencies**

- Ensure `mlx-lm` is installed (version 0.15.2 or later recommended)

- Run: `pip install mlx-lm`

2. **Load the Model**

- Import the required functions: `from mlx_lm import load, generate`

- Load the model and tokenizer: `model, tokenizer = load("mlx-community/Llama-3-Groq-8B-Tool-Use-4bit")`

- Note: First run will download the model files (~4.5GB for 4-bit quantized version)

3. **Generate Text**

- Call `generate(model, tokenizer, prompt=user_prompt, verbose=True)`

- The `verbose=True` flag provides token generation feedback

- For function calling, structure your prompt with tool definitions and user queries

4. **Customize Generation Parameters** (optional)

- `max_tokens`: Control output length (default varies)

- `temp`: Adjust temperature for creativity (0.0 = deterministic, 1.0 = creative)

- `top_p`: Nucleus sampling threshold

- `repetition_penalty`: Reduce repetitive outputs

5. **Function Calling Format**

- For tool use, format prompts with available functions/tools in the system message

- Follow Llama-3 chat template format for best results

- The model will generate structured function calls when appropriate

Example Usage

**Basic text generation:**

```python

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Llama-3-Groq-8B-Tool-Use-4bit")

response = generate(model, tokenizer, prompt="Explain quantum computing", verbose=True)

print(response)

```

**Function calling example:**

```python

prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You have access to the following functions:

<|eot_id|><|start_header_id|>user<|end_header_id|>

What's the weather in San Francisco?<|eot_id|><|start_header_id|>assistant<|end_header_id|>"""

response = generate(model, tokenizer, prompt=prompt, max_tokens=256)

```

Important Notes

Constraints