Skip to main content
Custom chunking allows you to implement your own chunking strategy by creating a class that inherits from ChunkingStrategy. This is useful when you need to split documents based on specific separators, apply custom logic, or handle domain-specific content formats.
1

Create a Python file

touch custom_chunking.py
2

Add the following code to your Python file

custom_chunking.py
from typing import List
import asyncio
from agno.agent import Agent
from agno.knowledge.chunking.base import ChunkingStrategy
from agno.knowledge.content import Document
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.pdf_reader import PDFReader
from agno.vectordb.pgvector import PgVector

class CustomChunking(ChunkingStrategy):
    def __init__(self, separator: str = "---", **kwargs):
        self.separator = separator

    def chunk(self, document: Document) -> List[Document]:
        # Split by custom separator
        chunks = document.content.split(self.separator)

        result = []
        for i, chunk_content in enumerate(chunks):
            chunk_content = self.clean_text(chunk_content)  # Use inherited method
            if chunk_content:
                meta_data = document.meta_data.copy()
                meta_data["chunk"] = i + 1
                result.append(Document(
                    id=f"{document.id}_{i+1}" if document.id else None,
                    name=document.name,
                    meta_data=meta_data,
                    content=chunk_content
                ))
        return result

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

knowledge = Knowledge(
    vector_db=PgVector(table_name="recipes_custom_chunking", db_url=db_url),
)

asyncio.run(knowledge.add_content_async(
    url="https://agno-public.s3.amazonaws.com/recipes/ThaiRecipes.pdf",
    reader=PDFReader(
        name="Custom Chunking Reader",
        chunking_strategy=CustomChunking(separator="---"),
    ),
))

agent = Agent(
    knowledge=knowledge,
    search_knowledge=True,
)

agent.print_response("How to make Thai curry?", markdown=True)
3

Create a virtual environment

Open the Terminal and create a python virtual environment.
python3 -m venv .venv
source .venv/bin/activate
4

Install libraries

pip install -U sqlalchemy psycopg pgvector agno
5

Run PgVector

docker run -d \
  -e POSTGRES_DB=ai \
  -e POSTGRES_USER=ai \
  -e POSTGRES_PASSWORD=ai \
  -e PGDATA=/var/lib/postgresql/data/pgdata \
  -v pgvolume:/var/lib/postgresql/data \
  -p 5532:5432 \
  --name pgvector \
  agno/pgvector:16
6

Run the script

python custom_chunking.py

​
Custom Chunking Params

ParameterTypeDefaultDescription
separatorstr"---"The string used to split the document content into chunks.