# Multimodal inputs (/use-cases/data-labeling/multimodal-inputs)



Pass media through the matching `Agent.run()` argument and choose a model that supports the input modality. The `output_schema` pattern stays the same.

<Note>
  `output_schema` requests structured output and Agno attempts to parse it. A failed or unparseable run can leave `content` as text. Check `RunStatus.completed` and the expected Pydantic type before reading fields, indexing, or writing downstream. These examples stop on failure; an application can instead send the original input to a review queue. Schema validation checks the shape and constraints, so verify extracted facts against the source separately.
</Note>

```python
from agno.run.base import RunStatus

from typing import Literal

from agno.agent import Agent
from agno.media import Image
from agno.models.google import Gemini
from pydantic import BaseModel, Field


class Classification(BaseModel):
    label: Literal["wildlife", "landscape", "sports", "architecture", "other"] = Field(
        ..., description="The primary scene type of the image"
    )


agent = Agent(
    model=Gemini(id="gemini-3.5-flash"),
    instructions="You classify images by scene type.",
    output_schema=Classification,
)

url = "https://storage.googleapis.com/generativeai-downloads/images/generated_elephants_giraffes_zebras_sunset.jpg"
result_run = agent.run("Classify this image.", images=[Image(url=url)])
if result_run.status != RunStatus.completed or not isinstance(result_run.content, Classification):
    raise RuntimeError("No validated Classification; send the input to review before continuing")
result = result_run.content
# Classification(label='wildlife')
```

## Input argument per modality [#input-argument-per-modality]

| Modality | Import                         | Argument                                    |
| -------- | ------------------------------ | ------------------------------------------- |
| Image    | `from agno.media import Image` | `images=[Image(url=...)]`                   |
| Audio    | `from agno.media import Audio` | `audio=[Audio(content=...)]`                |
| Video    | `from agno.media import Video` | `videos=[Video(content=..., format="mp4")]` |
| PDF      | `from agno.media import File`  | `files=[File(url=...)]`                     |

All four classes accept `url`, `filepath`, or raw bytes via `content`. The cookbook examples fetch audio and video bytes first. Agno installs `httpx`.

```python
from agno.run.base import RunStatus

import httpx

from agno.agent import Agent
from agno.media import Audio
from agno.models.google import Gemini
from pydantic import BaseModel, Field


class Transcript(BaseModel):
    text: str = Field(..., description="Verbatim transcript of all spoken audio")


transcription_agent = Agent(
    model=Gemini(id="gemini-3.5-flash"),
    instructions="Transcribe all spoken audio. Return only the transcript.",
    output_schema=Transcript,
)

url = "https://agno-public.s3.us-east-1.amazonaws.com/demo_data/QA-01.mp3"
response = httpx.get(url, timeout=30.0)
response.raise_for_status()

result_run = transcription_agent.run(
    "Transcribe this audio.",
    audio=[Audio(content=response.content)],
)
if result_run.status != RunStatus.completed or not isinstance(result_run.content, Transcript):
    raise RuntimeError("No validated Transcript; send the input to review before continuing")
result = result_run.content
# Transcript(text='...')
```

## Bounding boxes [#bounding-boxes]

For region detection, return normalized coordinates so the result is resolution-independent.

```python
from pydantic import BaseModel, Field


class BoundingBox(BaseModel):
    label: str = Field(..., description="What the box contains")
    x: float = Field(..., ge=0.0, le=1.0, description="Top-left x in [0, 1]")
    y: float = Field(..., ge=0.0, le=1.0, description="Top-left y in [0, 1]")
    width: float = Field(..., ge=0.0, le=1.0, description="Width in [0, 1]")
    height: float = Field(..., ge=0.0, le=1.0, description="Height in [0, 1]")
```

<Warning>
  State the `[0, 1]` coordinate system in both the field descriptions and the agent instructions. This keeps the four values consistent across images.
</Warning>

The [bounding boxes cookbook](https://github.com/agno-agi/agno/tree/main/cookbook/data_labeling/_08_image_bounding_boxes) has runnable single-object, multi-object, and per-box confidence variants.

## Transcription and diarization [#transcription-and-diarization]

Audio extraction covers transcription, speaker diarization, and timestamped segments. Each is a schema change over the same API.

| Output               | Schema shape                                               |
| -------------------- | ---------------------------------------------------------- |
| Flat transcript      | `{ text: str }`                                            |
| Speaker turns        | `{ turns: List[{ speaker, text }] }`                       |
| Timestamped segments | `{ segments: List[{ start_seconds, end_seconds, text }] }` |

## Model choice [#model-choice]

The cookbook uses `gemini-3.5-flash` across its text, image, audio, video, and PDF recipes. A replacement model must support the input modality and structured output.

## Next steps [#next-steps]

| Task                     | Guide                                                             |
| ------------------------ | ----------------------------------------------------------------- |
| Define the output schema | [Data extraction](/use-cases/data-labeling/structured-extraction) |
| Assign labels to media   | [Classification](/use-cases/data-labeling/classification)         |
| Review media labels      | [Quality pipeline](/use-cases/data-labeling/quality-pipeline)     |

## Developer Resources [#developer-resources]

* [Image extraction cookbook](https://github.com/agno-agi/agno/tree/main/cookbook/data_labeling/_07_image_extraction)
* [Audio transcription cookbook](https://github.com/agno-agi/agno/tree/main/cookbook/data_labeling/_11_audio_transcription)
* [Video extraction cookbook](https://github.com/agno-agi/agno/tree/main/cookbook/data_labeling/_14_video_extraction)
* [Document extraction cookbook](https://github.com/agno-agi/agno/tree/main/cookbook/data_labeling/_16_document_extraction)
* [Multimodal agents](/multimodal/overview)