Image OCR with Structured Output

Extract a structured bill schema from a receipt image with a Ministral 14B agent.

Code

from typing import List

from agno.agent import Agent
from agno.media import Image
from agno.models.mistral.mistral import MistralChat
from pydantic import BaseModel

class GroceryItem(BaseModel):
    item_name: str
    price: float

class GroceryListElements(BaseModel):
    bill_number: str
    items: List[GroceryItem]
    total_price: float

agent = Agent(
    model=MistralChat(id="ministral-14b-2512"),
    instructions=[
        "Extract the text elements described by the user from the picture",
    ],
    output_schema=GroceryListElements,
    markdown=True,
)

agent.print_response(
    "From this restaurant bill, extract the bill number, item names and associated prices, and total price and return it as a string in a Json object",
    images=[Image(url="https://i.imghippo.com/files/kgXi81726851246.jpg")],
)

output_schema describes the expected type. If parsing or validation fails, result.content can remain a string. Before accessing schema fields in a run result, use isinstance(result.content, YourSchema), replacing YourSchema with the class you passed as output_schema.

Usage

Set up your virtual environment

uv venv --python 3.12
source .venv/bin/activate

Set your API key

export MISTRAL_API_KEY=xxx

Install dependencies

uv pip install -U mistralai agno

Run Agent

Save the code above as image_ocr_with_structured_output.py, then run:

python image_ocr_with_structured_output.py