Data extraction

Extract typed Pydantic objects from text, images, audio, video, and PDFs.

Define the schema, pass the input, get a validated object back.

from typing import Optional

from agno.agent import Agent
from agno.models.google import Gemini
from pydantic import BaseModel, Field


class Contact(BaseModel):
    name: Optional[str] = Field(None, description="Full name as written")
    email: Optional[str] = Field(None, description="Email address")
    phone: Optional[str] = Field(None, description="Phone number, raw format")
    company: Optional[str] = Field(None, description="Company or organization")
    title: Optional[str] = Field(None, description="Job title")


agent = Agent(
    model=Gemini(id="gemini-3.5-flash"),
    instructions=(
        "Extract contact information from the input. Use exactly what the "
        "text shows. If a field is missing, leave it null. Do not guess."
    ),
    output_schema=Contact,
)

result = agent.run(
    "Hi - Sarah Johnson, VP of Marketing at Acme Corp. "
    "[e-mail] / [phone]."
).content
# Contact(name='Sarah Johnson', email='[e-mail]',
#         phone='[phone]', company='Acme Corp.', title='VP of Marketing')

Tell the agent how to handle missing fields: “If a field is missing, leave it null. Do not guess.”

Nested objects

Lists of sub-objects work the same way. Define the inner model and reference it.

from typing import List, Optional

from pydantic import BaseModel, Field


class ActionItem(BaseModel):
    owner: str = Field(..., description="Person responsible")
    description: str = Field(..., description="What needs to be done")
    due_date: Optional[str] = Field(None, description="Due date if stated")


class Meeting(BaseModel):
    action_items: List[ActionItem] = Field(default_factory=list)

Pass output_schema=Meeting and the agent returns a Meeting with a populated action_items list.

Per-field confidence

When downstream needs to route uncertain fields to a human, wrap each value in a confidence carrier.

from typing import Literal, Optional

from agno.agent import Agent
from agno.models.google import Gemini
from pydantic import BaseModel, Field


class ConfidentField(BaseModel):
    value: Optional[str] = None
    confidence: Literal["high", "medium", "low"] = Field(
        ..., description="Confidence in the extracted value"
    )


class Contact(BaseModel):
    name: ConfidentField
    email: ConfidentField
    company: ConfidentField


instructions = """\
Extract contact information from the input. For each field:
- value:      what the text shows; null if the field is missing
- confidence: high if explicit and unambiguous;
              medium if implied or partially formatted;
              low if guessed or ambiguous

Use exactly what the text shows. Do not normalize or paraphrase.
"""

confidence_agent = Agent(
    model=Gemini(id="gemini-3.5-flash"),
    instructions=instructions,
    output_schema=Contact,
)

Define each confidence level in the instructions so labels use the same criteria across records.

Any modality, same pattern

The input argument changes per modality. Use a model that supports both that modality and structured output. See Multimodal inputs for the input arguments.

Input	Argument	Cookbook
Text	agent.run(text)	text_extraction
Image	images=[Image(url=...)]	image_extraction
Audio	audio=[Audio(content=...)]	audio_extraction
Video	videos=[Video(content=..., format="mp4")]	video_extraction
PDF	files=[File(url=...)]	document_extraction

Extract then index

The image extraction to vector database cookbook is the minimal pipeline. It describes each image as a typed object, flattens the object into searchable text, embeds it, and stores it in LanceDb.

This recipe needs LanceDb on top of the Google provider:

uv pip install lancedb tantivy

The Image Search application expands the same pattern into an ingest workflow, PgVector hybrid search, AgentOS endpoints, and a browser UI.

Next steps

Task	Guide
Assign labels	Classification
Build a searchable image library	Image Search
Feed non-text input	Multimodal inputs
Add a reviewer and adjudicator	Quality pipeline

Developer Resources

• Text extraction cookbook
• Structured output