Claude Vision API: Image Analysis At Production Scale

Vision Looks Like Magic Until You Run 100k Images Through It

The first time you hand Claude a screenshot and it correctly extracts every field on a messy invoice, the temptation is to stick that in a loop and bill your way to a finished product. Then the bill arrives, half your images came back with subtle hallucinations, and you discover the support team has been emailing you screenshots that are actually photos of laptop screens with glare on them.

Claude's vision API is genuinely strong on structured imagery - documents, charts, diagrams, UI screenshots, code on screens. It is genuinely weak on the failure modes you only notice in production: low-resolution thumbnails, handwriting, rotated photos, images where the relevant content is 5% of the pixels. This guide covers the production patterns we use to ship vision pipelines that actually hold up at volume.

We walked through some of the more counterintuitive tricks in our Vision API Tricks: Extract Data from Screenshots video on YouTube. This post is the long-form companion.

What Claude Vision Is Good At, And What It Isn't

Claude reliably crushes:

• Structured documents - invoices, receipts, tax forms, lab reports, anything with consistent layout
• Charts and graphs - extracting series, axis labels, trend descriptions
• UI screenshots - element identification, design feedback, accessibility audits
• Code-in-images - copying code from a screenshot back into text with high fidelity
• Diagrams - flowcharts, architecture diagrams, ER diagrams
• Tables - even slightly skewed, multi-column tables come back clean

Claude struggles with, in our experience:

• Handwriting - better than it used to be, still unreliable on anything cursive
• Photos of screens - the moire patterns and reflections murder accuracy
• Tiny detail - anything smaller than ~12 pixels of feature size is a coin flip
• Counting - "how many people are in this image" is genuinely hard for every vision model
• Spatial reasoning in cluttered scenes - "is the cup to the left of the keyboard"

Treat the second list as warning signs, not deal-breakers. With the right preprocessing and prompting, most of them get usable. Without it, you ship hallucinations.

Image Input: Formats, Limits, And The Cost-Per-Pixel Reality

Vision pricing is token-based, but the token count is computed from the image dimensions. The exact formula is roughly tokens ~ (width x height) / 750, capped after Anthropic's resizing logic. Two practical consequences:

1. A 2000x2000 image costs about 5300 tokens of input. That is not free.
2. Resizing is your most powerful cost lever. Most "high-resolution" images can be downscaled to 1024 pixels on the long edge with zero quality loss for documents and screenshots.

Supported formats are JPEG, PNG, GIF, and WebP. You can pass a URL or base64. URL is preferable when you have a CDN; base64 is fine for pipelines that already have the bytes in memory.

For most production workloads, the right preprocessing pipeline is:

1. Decode and auto-orient via EXIF
2. Convert to RGB
3. Downscale longest edge to 1568 pixels (Anthropic's preferred max)
4. Re-encode as JPEG quality 85
5. Hash the result for cache deduplication

This typically cuts image costs 40 to 70% with no measurable accuracy loss on text-heavy imagery.

The TypeScript Code You Should Actually Ship

Here is a minimal but production-shaped vision call using the official Anthropic SDK. It accepts a base64 image, asks for structured extraction, and uses prompt caching on the instruction prefix so a high-volume pipeline does not pay full input cost on every call.

import Anthropic from "@anthropic-ai/sdk";
import sharp from "sharp";

const client = new Anthropic();

async function preprocessImage(buf: Buffer): Promise<{ data: string; mediaType: "image/jpeg" }> {
  const out = await sharp(buf)
    .rotate()
    .resize({ width: 1568, height: 1568, fit: "inside", withoutEnlargement: true })
    .jpeg({ quality: 85 })
    .toBuffer();
  return { data: out.toString("base64"), mediaType: "image/jpeg" };
}

const EXTRACTION_PROMPT = `You are a precise document extractor. Return only valid JSON matching:
{ "vendor": string, "date": string, "total": number, "line_items": [{"description": string, "amount": number}] }
If a field is unreadable, use null. Do not invent values.`;

export async function extractInvoice(imageBytes: Buffer) {
  const { data, mediaType } = await preprocessImage(imageBytes);

  const response = await client.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    system: [
      {
        type: "text",
        text: EXTRACTION_PROMPT,
        cache_control: { type: "ephemeral" },
      },
    ],
    messages: [
      {
        role: "user",
        content: [
          {
            type: "image",
            source: { type: "base64", media_type: mediaType, data },
          },
          { type: "text", text: "Extract this invoice." },
        ],
      },
    ],
  });

  const text = response.content.find((b) => b.type === "text");
  if (!text || text.type !== "text") throw new Error("no text response");
  return JSON.parse(text.text);
}

A few non-obvious things this shape captures:

• The instructions are in system with cache_control, not in the user turn. Vision pipelines tend to have a stable instruction and a varying image; cache the stable part.
• The image goes inside the user turn alongside a short text prompt. The text is what cues the model on what to do; without it, you get a generic description.
• The response is parsed strictly. Vision models are slightly more prone to wrapping JSON in markdown - you can add a <json> tag instruction or use a tool-use schema to lock it down.

For schema-locked extraction, use tool use. Define a tool with the exact JSON Schema you want, force tool_choice to that tool, and you get back a guaranteed-valid object. We covered the details in tool use patterns.

Production Use Cases Where Vision Pays For Itself

1. Invoice and receipt OCR. Vision plus a strict schema delivers extraction accuracy in the high 90s on real-world receipts - better than most dedicated OCR services on messy inputs, because Claude can reason about layout. Cost runs about $0.005 to $0.015 per receipt depending on resolution.

2. Bug-report screenshot triage. Customer pastes a screenshot, vision extracts the visible error message, the URL bar, the browser, and the visible state. Auto-tagged into the issue tracker. We have seen support teams cut triage time by 60% on this single workflow.

3. UI accessibility audits. Feed a screenshot of a page, ask for WCAG violations. Claude is shockingly good at "this contrast looks bad," "the touch target is too small," "this label is missing for the icon." Not a replacement for axe-core, but an excellent complement on visual issues automated tools miss.

4. Chart-to-data extraction. Bar charts, line charts, scatter plots - Claude returns reasonable JSON of the underlying series. Watch out: numbers are estimates derived from pixels, not ground truth. Use it for "what is this chart showing," not for downstream analytics.

5. Diagram-to-code. Hand-drawn architecture diagrams or flowcharts converted to mermaid, plantuml, or actual code stubs. Underrated workflow for design reviews.

Building A Reliable Vision Pipeline

Volume changes everything. At one image per minute you can ignore the failure modes; at 100k a month they dominate.

The pipeline shape we have ended up with after several iterations:

[ingest] -> [validate] -> [preprocess] -> [hash + cache lookup]
       -> [vision call w/ retries] -> [schema validate]
       -> [confidence gate] -> [human review or auto-accept]
       -> [store + index]

Each step earns its keep:

• Validate. Reject inputs that aren't actually images, are too small (under 200px on a side is a guarantee of garbage), or are corrupt. About 1 to 3% of inputs fail this gate in real apps.
• Hash + cache lookup. A surprising fraction of "different" images are the same file uploaded twice. Hash the preprocessed bytes and skip the API call on duplicates. Easy 10 to 25% cost reduction.
• Schema validate. Run the JSON response through Zod or your validator. About 0.5 to 2% of responses fail the schema even with strict prompting; retry once with a stricter prompt.
• Confidence gate. Have the model return a confidence field, and route low-confidence outputs to human review. This is the single highest-impact reliability lever.

For monitoring spend across a high-volume vision pipeline, CodeBurn surfaces per-route token cost so you can tell instantly when an image-preprocessing regression doubles your bill.

Handling Errors And Edge Cases

The errors you will actually hit:

• invalid_request_error: image too large - You sent over 5MB or over 8000px on a side. Preprocess.
• overloaded_error - Vision endpoints get bursty. Exponential backoff with jitter. Three retries is enough.
• Truncated JSON - Set max_tokens high enough. For invoice extraction, 2000 is safer than 1024.
• Confident hallucination - The model returns a valid-looking number that doesn't exist in the image. The defense is a confidence-gated human review for any field used in money decisions.
• Mixed-language documents - Claude handles them but sometimes returns translated values when you wanted the original. Always specify "return values exactly as printed, do not translate."

Multi-Image Analysis And Sequence Reasoning

You can pass multiple images in a single user turn. The model treats them as an ordered sequence. Common uses:

• Before/after comparisons - "what changed between these two screenshots"
• Multi-page documents - pass all pages of a PDF as separate images for end-to-end extraction
• Product image grids - analyze a sheet of variants in one call

Cost scales linearly with images. A four-image call costs roughly four times a one-image call (plus a small fixed overhead). For long documents, multi-image extraction in a single call is usually cheaper and more accurate than calling once per page, because the model can reason across pages.

Production Gotchas Worth Pinning To Your Wall

1. EXIF orientation is not auto-applied. A photo taken in portrait will land at the API rotated unless you bake the orientation into the bytes. We have seen entire pipelines fail because the iPhone landscape vs portrait flag was ignored.

2. Animated GIFs are sampled, not analyzed frame-by-frame. You typically get the first frame. If you need video analysis, pass key frames as separate images.

3. The model will describe what it sees if you don't tell it not to. Open-ended prompts like "what is this" produce flowery prose. For extraction, be explicit: return JSON only, no commentary.

4. Image content counts against the cache. A cached system prompt plus an uncached image is the right pattern. You cannot meaningfully cache the image itself unless the exact same bytes recur, in which case do it via hash dedup at your layer.

5. PII in images is a real compliance issue. Vision will happily extract SSNs, account numbers, faces. If you process user-uploaded images, run a redaction or detection pass and have an explicit data retention policy. Anthropic's data is not used for training by default on the API, but your own logs probably retain images longer than you think.

6. Resolution requirements vary by task. UI screenshots can be downscaled aggressively. Tiny text in a fax-quality scan needs the full resolution. Don't blanket-downscale everything; route by content type.

Scaling Vision At Production Volume

For 100k+ images a month, three things matter more than the SDK call:

• Concurrency limits. Anthropic's vision rate limits are tight. Use a token bucket and aim for ~80% of your stated limit. Bursting up to the limit causes more 529s than it earns in throughput.
• Async architecture. Most vision workloads are not user-facing. Push them through a queue, return a job ID, notify on completion. For the truly batchable, the Batch API cuts cost in half.
• Cost attribution. Tag every call with the originating product and route. We have watched a single buggy feature triple a vision bill in a week because no one had per-feature attribution. The 400-Dollar Overnight Bill post-mortem walks through what bad attribution costs.

Production Checklist Before You Ship

• Image preprocessing: orient, resize to 1568px, JPEG q85
• Hash-based dedup cache in front of the API
• System prompt cached with cache_control
• Tool-use schema for any structured extraction
• Schema validation on every response
• Confidence field returned and gated for human review
• Retry with backoff on overloaded errors
• Per-feature token attribution logged
• PII detection and retention policy on uploaded images
• Async queue for non-interactive workloads
• Alert on cost-per-image regressions

Vision is the most under-used feature in the Claude API among teams that already have text working. The ones who get it right at scale treat it as an industrial pipeline, not a model call. Preprocess, cache, validate, attribute, and you will ship a vision feature that holds up.

For more on shipping Claude in production, see our writeups on prompt caching and tool use patterns.