Deploy Llama 2, Mistral, Yi, Zephyr, or Deepseek models with structured function calling capabilities. Returns JSON responses with function names and arguments.
This skill has been flagged as potentially dangerous. It contains patterns that could compromise your security or manipulate AI behavior.Safety score: 20/100.
KillerSkills scans all public content for safety. Use caution before installing or executing flagged content.
Deploy open-source language models (Llama 2, Mistral, Yi, Zephyr, Deepseek Coder) with structured function calling capabilities. These models respond with JSON objects containing function names and arguments, enabling seamless integration with external tools and APIs.
Guides you through setting up and using fine-tuned open-source models that understand function calling syntax. The models accept function descriptions and user prompts, then return structured JSON responses indicating which function to call and with what parameters.
**Free:**
**Paid (purchase from Trelis):**
Select one of these options:
**Option A: Text Generation Inference (Recommended for Production)**
1. Clone the setup repository: `git clone https://github.com/TrelisResearch/tgi-chat-ui-function-calling`
2. Follow the README to deploy with Docker
3. Access via API at `http://localhost:8080`
**Option B: RunPod (Cloud GPU)**
1. Use the RunPod template: https://runpod.io/gsc?template=edxvuji38p&ref=jmfkcdio
2. Once running, your endpoint will be: `https://{YOUR_POD_ID}-8080.proxy.runpod.net`
**Option C: Local with llama.cpp**
1. Download GGUF model file from HuggingFace
2. Run llama.cpp server: `./server -m model.gguf --host 0.0.0.0 --port 8080`
3. Access at `http://localhost:8080`
Create function metadata in JSON format:
```python
function_metadata = {
"function": "search_bing",
"description": "Search the web for content on Bing. This allows users to search online/the internet/the web for content.",
"arguments": [
{
"name": "query",
"type": "string",
"description": "The search query string"
}
]
}
```
Supported argument types: `string`, `number`, `array`
Use the correct prompt template for your model:
```python
import json
B_FUNC, E_FUNC = "<FUNCTIONS>", "</FUNCTIONS>\n\n"
B_INST, E_INST = "[INST] ", " [/INST]" # Llama/Mistral/Zephyr
function_list = json.dumps(function_metadata, indent=4)
user_prompt = 'Search for the latest news on AI.'
prompt = f"{B_FUNC}{function_list.strip()}{E_FUNC}{B_INST}{user_prompt.strip()}{E_INST}\n\n"
```
**With system message:**
```python
B_SYS, E_SYS = "<<SYS>>\n", "\n<</SYS>>\n\n"
system_prompt = "You are a helpful assistant."
prompt = f"{B_FUNC}{function_list.strip()}{E_FUNC}{B_INST}{B_SYS}{system_prompt.strip()}{E_SYS}{user_prompt.strip()}{E_INST}\n\n"
```
```python
import requests
url = "http://localhost:8080/completion" # or your RunPod URL
response = requests.post(url, json={"prompt": prompt})
result = response.json()
print(result)
```
The model returns JSON in this format:
```json
{
"function": "search_bing",
"arguments": {
"query": "latest AI news"
}
}
```
**Always validate:**
Extract the function name and arguments, then call your actual implementation:
```python
import json
response_text = result['content']
try:
function_call = json.loads(response_text)
function_name = function_call['function']
arguments = function_call['arguments']
# Call your actual function
if function_name == "search_bing":
result = search_bing(arguments['query'])
except json.JSONDecodeError:
# Handle non-JSON responses
print("Model did not return valid JSON")
```
Larger models have better training losses: 0.5 (7B), 0.4 (13B), 0.3 (70B).
1. **Write clear function descriptions** — Include whether arguments are required and default values
2. **Validate user input** — Post-process responses to ensure all required info is provided
3. **Handle missing data gracefully** — Prompt users for missing information rather than failing silently
4. **Test with multiple functions** — Models were trained on 0-3 arguments; test edge cases
5. **Use arrays when appropriate** — Models support array arguments like `["file1.txt", "file2.txt"]`
```python
functions = [
{
"function": "search_bing",
"description": "Search the web for content.",
"arguments": [{"name": "query", "type": "string", "description": "Search query"}]
},
{
"function": "search_arxiv",
"description": "Search for research papers. Use AND, OR, NOT operators.",
"arguments": [{"name": "query", "type": "string", "description": "Search query"}]
},
{
"function": "delete_file",
"description": "Delete one or more files.",
"arguments": [{"name": "fileNames", "type": "array", "description": "List of file names"}]
}
]
function_list = "\n\n".join([json.dumps(f, indent=4) for f in functions])
```
**Model returns text instead of JSON:**
**Missing arguments:**
**Performance issues:**
All Llama models subject to [Meta license terms](https://ai.meta.com/resources/models-and-libraries/llama-downloads/).
Leave a review
No reviews yet. Be the first to review this skill!
# Download SKILL.md from killerskills.ai/api/skills/llama-2-function-calling/raw