Trafilatura - Agno

TrafilaturaTools enable an Agent to extract text and metadata from web pages using the Trafilatura library.

Prerequisites

The following example requires the trafilatura library.

uv pip install -U trafilatura openai

Example

The following agent can extract and analyze web content:

from agno.agent import Agent
from agno.tools.trafilatura import TrafilaturaTools

agent = Agent(
    instructions=[
        "You are a web content extraction specialist",
        "Extract clean text and structured data from web pages",
        "Provide detailed analysis of web content and metadata",
        "Help with content research and web data collection",
    ],
    tools=[TrafilaturaTools()],
)

agent.print_response("Extract the main content from https://example.com/article", stream=True)

Toolkit Params

Parameter	Type	Default	Description
`output_format`	`str`	`"txt"`	Default output format (txt, json, xml, markdown, csv, html, xmltei).
`include_comments`	`bool`	`True`	Whether to extract comments along with main text.
`include_tables`	`bool`	`True`	Whether to include table content.
`include_images`	`bool`	`False`	Whether to include image information (experimental).
`include_formatting`	`bool`	`False`	Whether to preserve text formatting.
`include_links`	`bool`	`False`	Whether to preserve links (experimental).
`with_metadata`	`bool`	`False`	Whether to include metadata in extractions.
`favor_precision`	`bool`	`False`	Whether to prefer precision over recall.
`favor_recall`	`bool`	`False`	Whether to prefer recall over precision.
`target_language`	`Optional[str]`	`None`	Target language filter (ISO 639-1 format).
`deduplicate`	`bool`	`False`	Whether to remove duplicate segments.
`max_tree_size`	`Optional[int]`	`None`	Maximum tree size for processing.
`max_crawl_urls`	`int`	`10`	Maximum number of URLs to crawl per website.
`max_known_urls`	`int`	`100000`	Maximum number of known URLs during crawling.
`enable_extract_text`	`bool`	`True`	Whether to extract text content.
`enable_extract_metadata_only`	`bool`	`True`	Whether to extract metadata information.
`enable_html_to_text`	`bool`	`True`	Whether to convert HTML content to clean text.
`enable_extract_batch`	`bool`	`True`	Whether to extract content from multiple URLs in batch.
`enable_crawl_website`	`bool`	`True`	Whether to crawl a website and extract content from multiple pages.
`all`	`bool`	`False`	Enable all tools.

Toolkit Functions

Function	Description
`extract_text`	Extract main text content from a URL.
`extract_metadata_only`	Extract metadata information from web pages.
`html_to_text`	Convert HTML content to clean text.
`crawl_website`	Crawl a website and extract content from multiple pages.
`extract_batch`	Extract content from multiple URLs in batch.

Developer Resources