Vision (Multi-modal) — Hypery Docs

Overview

Vision-capable models can analyze images alongside text, enabling applications like image description, OCR, visual question answering, chart analysis, and more.

Authentication: API Key or OAuth token
Required Scopes (OAuth): ai:chat

Capabilities

Image Description - Get detailed descriptions of images
OCR - Extract text from screenshots and documents
Visual Q&A - Answer questions about image content
Comparison - Analyze multiple images together
Chart Analysis - Extract insights from visualizations

Supported Models

Model	Quality	Speed	Best For
`anthropic/claude-3.5-sonnet`	Excellent	Fast	General vision tasks
`anthropic/claude-3-opus`	Best	Medium	Complex analysis
`openai/gpt-4o`	Excellent	Fast	Real-time vision
`google/gemini-pro-vision`	Excellent	Fast	High-volume

Analyze Image

Send images with text prompts for analysis.

Endpoint: POST /api/v1/chat/completions

import OpenAI from 'openai'
 
const client = new OpenAI({
  apiKey: process.env.GATEWAY_API_KEY,
  baseURL: 'https://hypery.ai/api/v1'
})
 
const response = await client.chat.completions.create({
  model: 'anthropic/claude-3.5-sonnet',
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'What is in this image?' },
        {
          type: 'image_url',
          image_url: {
            url: 'https://example.com/photo.jpg'
          }
        }
      ]
    }
  ],
  max_tokens: 500
})
 
console.log(response.choices[0].message.content)

from openai import OpenAI
 
client = OpenAI(
    api_key="your-api-key",
    base_url="https://hypery.ai/api/v1"
)
 
response = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://example.com/photo.jpg"
                    }
                }
            ]
        }
    ],
    max_tokens=500
)
 
print(response.choices[0].message.content)

curl https://hypery.ai/api/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-3.5-sonnet",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "What is in this image?"},
          {
            "type": "image_url",
            "image_url": {"url": "https://example.com/photo.jpg"}
          }
        ]
      }
    ],
    "max_tokens": 500
  }'

Input Parameters

Parameter	Type	Required	Description
`model`	string	Yes	Vision-capable model ID
`messages`	array	Yes	Messages with text and image content
`messages[].content`	array	Yes	Array of content parts (text and images)
`max_tokens`	number	No	Maximum tokens (recommended: 500-1000 for vision)

Content Types

Type	Format	Description
`text`	`{type: "text", text: "..."}`	Text prompt or question
`image_url`	`{type: "image_url", image_url: {url: "..."}}`	Image URL or base64 data URI

Response:

{
  "id": "chatcmpl-123",
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "This image shows a sunset over a beach with palm trees..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 1254,
    "completion_tokens": 87,
    "total_tokens": 1341
  }
}

Images are converted to tokens based on their resolution. Higher resolution images use more tokens and cost more.

Image Formats

URL Reference

Use publicly accessible image URLs:

{
  type: 'image_url',
  image_url: {
    url: 'https://example.com/photo.jpg'
  }
}

{
    "type": "image_url",
    "image_url": {
        "url": "https://example.com/photo.jpg"
    }
}

Base64 Data URI

Encode images directly in the request:

const fs = require('fs')
 
const imageBuffer = fs.readFileSync('photo.jpg')
const base64Image = imageBuffer.toString('base64')
 
{
  type: 'image_url',
  image_url: {
    url: `data:image/jpeg;base64,${base64Image}`
  }
}

import base64
 
with open("photo.jpg", "rb") as image_file:
    base64_image = base64.b64encode(image_file.read()).decode('utf-8')
 
{
    "type": "image_url",
    "image_url": {
        "url": f"data:image/jpeg;base64,{base64_image}"
    }
}

Supported Formats

JPEG/JPG - Recommended for photos
PNG - Best for screenshots, graphics with text
WebP - Good compression, widely supported
GIF - First frame only (no animation)

Size Limits

Limit	Value
Max file size (base64)	20 MB
Max file size (URL)	50 MB
Recommended size	< 5 MB for best performance
Optimal resolution	512-2048px

Multiple Images

Analyze several images in one request:

const response = await client.chat.completions.create({
  model: 'anthropic/claude-3.5-sonnet',
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Compare these two images' },
        {
          type: 'image_url',
          image_url: { url: 'https://example.com/before.jpg' }
        },
        {
          type: 'image_url',
          image_url: { url: 'https://example.com/after.jpg' }
        }
      ]
    }
  ]
})

response = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Compare these two images"},
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/before.jpg"}
                },
                {
                    "type": "image_url",
                    "image_url": {"url": "https://example.com/after.jpg"}
                }
            ]
        }
    ]
)

Common Use Cases

Image Description

Get detailed descriptions of images:

const response = await client.chat.completions.create({
  model: 'anthropic/claude-3.5-sonnet',
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'Provide a detailed description of this image'
        },
        {
          type: 'image_url',
          image_url: { url: imageUrl }
        }
      ]
    }
  ],
  max_tokens: 500
})

response = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Provide a detailed description of this image"
                },
                {
                    "type": "image_url",
                    "image_url": {"url": image_url}
                }
            ]
        }
    ],
    max_tokens=500
)

OCR (Extract Text)

Extract text from images and documents:

const response = await client.chat.completions.create({
  model: 'anthropic/claude-3.5-sonnet',
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Extract all text from this image' },
        { type: 'image_url', image_url: { url: screenshotUrl } }
      ]
    }
  ]
})

response = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Extract all text from this image"},
                {"type": "image_url", "image_url": {"url": screenshot_url}}
            ]
        }
    ]
)

Chart Analysis

Extract insights from charts and graphs:

const response = await client.chat.completions.create({
  model: 'anthropic/claude-3.5-sonnet',
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'Analyze this chart and provide key insights'
        },
        {
          type: 'image_url',
          image_url: { url: chartUrl }
        }
      ]
    }
  ],
  max_tokens: 1000
})

response = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Analyze this chart and provide key insights"
                },
                {"type": "image_url", "image_url": {"url": chart_url}}
            ]
        }
    ],
    max_tokens=1000
)

Visual Q&A

Answer specific questions about images:

const response = await client.chat.completions.create({
  model: 'openai/gpt-4o',
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'How many people are in this photo?' },
        { type: 'image_url', image_url: { url: photoUrl } }
      ]
    }
  ]
})

response = client.chat.completions.create(
    model="openai/gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "How many people are in this photo?"},
                {"type": "image_url", "image_url": {"url": photo_url}}
            ]
        }
    ]
)

Multi-turn Conversations

Images are remembered throughout the conversation:

const messages = [
  {
    role: 'user',
    content: [
      { type: 'text', text: 'What is in this image?' },
      { type: 'image_url', image_url: { url: imageUrl } }
    ]
  }
]
 
// First response
let response = await client.chat.completions.create({
  model: 'anthropic/claude-3.5-sonnet',
  messages
})
 
messages.push(response.choices[0].message)
 
// Follow-up question - image is remembered
messages.push({
  role: 'user',
  content: 'What color is the car?'
})
 
response = await client.chat.completions.create({
  model: 'anthropic/claude-3.5-sonnet',
  messages
})

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What is in this image?"},
            {"type": "image_url", "image_url": {"url": image_url}}
        ]
    }
]
 
# First response
response = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=messages
)
 
messages.append(response.choices[0].message)
 
# Follow-up - image is remembered
messages.append({
    "role": "user",
    "content": "What color is the car?"
})
 
response = client.chat.completions.create(
    model="anthropic/claude-3.5-sonnet",
    messages=messages
)

Best Practices

Optimize Image Size

Resize images before sending to reduce cost:

// Most models work well with 1024x1024 or smaller
// Larger images = more tokens = higher cost

Be Specific in Prompts

// ❌ Vague
{ type: 'text', text: 'Describe this' }
 
// ✅ Specific
{
  type: 'text',
  text: 'Describe the architectural style, materials, and notable design features visible in this building'
}

Set Appropriate max_tokens

Vision responses can be detailed:

const response = await client.chat.completions.create({
  model: 'anthropic/claude-3.5-sonnet',
  messages: [...],
  max_tokens: 1000  // Allow detailed analysis
})

Handle Errors

try {
  const response = await client.chat.completions.create({
    model: 'anthropic/claude-3.5-sonnet',
    messages: [...]
  })
} catch (error) {
  if (error.message?.includes('image')) {
    console.error('Failed to load image:', error)
  } else if (error.status === 400) {
    console.error('Invalid image format or size')
  }
}

Next Steps

Function Calling

Combine vision with function calling

Streaming

Stream vision responses

Chat API

Back to Chat API overview