Computer vision solutions · 10–15%

Multimodal understanding

~2 min read

Vision-enabled chat

A multimodal model accepts more than one input type — here, images inside a text prompt — and reasons over both.

• Needs a multimodal model: GPT-4.1 / 4.1-mini, Phi-4-multimodal-instruct.
• Prompt = multi-part user message: input_text + input_image (Responses API) or text + image_url (ChatCompletions).
• Image via web URL or base64 data URL (data:image/jpeg;base64,…).
• Use cases the exam names: captions (concise or detailed, single or multiple images), visual question-answering grounded in the image, alt-text / extended descriptions for accessibility, interpreting charts, damage assessment.

{"role": "user", "content": [
    {"type": "input_text", "text": "What desserts could I make with this?"},
    {"type": "input_image", "image_url": data_url}
]}

Azure Content Understanding — the multimodal extractor

Azure Content Understanding is a Foundry Tool that uses generative AI to turn unstructured content into structured data. You define a schema describing the fields you want; the service extracts them from documents, images, audio, or video — one service for all four.

The pipeline, in order: you send the input; an analyzer applies your schema; content extraction runs OCR, transcription, and layout analysis; field extraction fills your schema fields; every value gets a confidence score and grounding; the output comes back as Markdown or JSON.

Field extraction methods — memorize

Key facts

• Prebuilt analyzers: prebuilt-image, prebuilt-receipt, prebuilt-invoice, prebuilt-idDocument.
• Custom analyzers: your schema, built in Content Understanding Studio (custom creation not in Foundry portal), minimal training data — schema by example.
• Confidence 0–1: ≥0.9 automate · 0.7–0.9 review if critical · <0.7 manual. Route low-confidence to humans.
• Grounding = source region where each value was found.
• The API is async: POST to :analyze (URL input) or analyzeBinary (raw bytes) starts the job and returns an Operation-Id; then GET analyzerResults/{id} repeatedly until the operation finishes.
• Image formats: JPEG, PNG, BMP, TIFF, HEIF, PDF.
• Video analysis: segments, summaries, activity detection; audio: call summaries, sentiment.
• Single-task vs pro mode: standard = one analyzer, one schema pass; pro mode = multi-step reasoning across multiple inputs/documents (e.g., cross-validating content).