Function calling & vision
Let models request your functions and understand images.
On this page
The function-calling loopChat Completions function definitionOther API formatsSend an image to a text modelThe function-calling loop
- Send function definitions to a model that supports tools.
- Inspect the returned function name, call ID, and arguments.
- Validate arguments and execute the function in your own application.
- Append the assistant call and the tool result to the conversation.
- Request the next completion; repeat until the model returns its final answer.
The model API does not execute your functions. Dedicated tools such as web search are separate HTTP operations that your application can call inside this loop. Validate model-provided arguments and apply your application permissions before executing actions.
Chat Completions function definition
Add this tools array to a chat request. On continuation, preserve the assistant message containing tool_calls and add a tool message with the matching tool_call_id.
{
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get weather for a city",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string"
}
},
"required": [
"city"
]
}
}
}
],
"tool_choice": "auto"
}Other API formats
Messages uses tool definitions with input_schema, assistant tool_use blocks, and user tool_result blocks with matching tool_use_id. Responses uses function definitions with top-level name and parameters, then function_call_output items with the original call_id. Preserve conversation context when submitting results.
Send an image to a text model
Choose a vision-capable text model. For Chat Completions, use a content array containing text and image_url blocks. Use a reachable image URL or a base64 data URL. Image understanding is separate from image generation.
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image."
},
{
"type": "image_url",
"image_url": {
"url": "https://YOUR_PUBLIC_HOST/image.png"
}
}
]
}