The moment our team at Presight AI shifted from building traditional RAG pipelines to agentic AI systems, everything changed. Our accuracy on complex analytical queries jumped from 62% to 89%, latency on multi-step tasks dropped by half, and—most importantly—our users stopped complaining that the AI "doesn't actually do anything useful."
This post is a practical guide distilled from over a year of building enterprise agentic systems in production. I will walk through the architecture decisions, code patterns, and hard-won lessons that made our agents reliable enough for regulated industries.
Why Agentic AI? The Limits of Traditional RAG
Retrieval-Augmented Generation works well for simple question-answering, but enterprise workflows rarely involve a single lookup. Consider a typical analyst request: "Compare Q3 revenue across our top 5 clients, flag anomalies, and draft an executive summary."
A traditional RAG pipeline retrieves documents and hopes the LLM can synthesize everything in one shot. An agentic system, by contrast, plans, acts, observes, and iterates—just like a human analyst would.
When Agents Beat RAG
- Multi-step reasoning: Tasks requiring more than one retrieval or computation
- Tool use: Querying databases, running calculations, calling APIs
- Self-correction: Detecting when an intermediate result is wrong and retrying
- Dynamic workflows: The next step depends on the output of the previous step
In our production systems at Presight AI, we found that roughly 40% of real user queries require at least two tool calls to answer properly. That is the sweet spot where agentic architectures pay for themselves.
Architecture: LangGraph for Multi-Agent Orchestration
After evaluating several frameworks—CrewAI, AutoGen, raw LangChain agents—we settled on LangGraph as our primary orchestration layer. The reason is simple: LangGraph gives you explicit control over the agent's state machine, which is non-negotiable in enterprise settings where you need auditability and deterministic routing.
Here is the skeleton of a production agent built with LangGraph:
from langgraph.graph import StateGraph, END
from langgraph.prebuilt import ToolNode
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage
from typing import TypedDict, Annotated, Sequence
import operator
class AgentState(TypedDict):
messages: Annotated[Sequence, operator.add]
next_action: str
iteration_count: int
def create_enterprise_agent():
llm = ChatOpenAI(
model="gpt-4o",
temperature=0,
max_retries=3,
)
tools = [query_database, run_analysis, search_documents]
llm_with_tools = llm.bind_tools(tools)
def reasoning_node(state: AgentState) -> AgentState:
"""Core reasoning: decide what to do next."""
messages = state["messages"]
response = llm_with_tools.invoke(messages)
return {
"messages": [response],
"iteration_count": state.get("iteration_count", 0) + 1,
}
def should_continue(state: AgentState) -> str:
"""Route: use a tool, or finish."""
last_message = state["messages"][-1]
if state.get("iteration_count", 0) > 10:
return "end" # Safety guardrail
if last_message.tool_calls:
return "tools"
return "end"
graph = StateGraph(AgentState)
graph.add_node("reason", reasoning_node)
graph.add_node("tools", ToolNode(tools))
graph.set_entry_point("reason")
graph.add_conditional_edges("reason", should_continue, {
"tools": "tools",
"end": END,
})
graph.add_edge("tools", "reason")
return graph.compile()
A few things to notice. First, the iteration guard (iteration_count > 10) is critical. Without it, a confused agent can loop forever, burning tokens and patience. Second, we keep temperature at zero for tool-calling agents—creativity is the enemy of reliability when you are deciding which SQL query to run.
Designing Tools That Agents Can Actually Use
The quality of your tools determines the ceiling of your agent's performance. This is the single most underrated aspect of building agentic AI systems. A poorly described tool will be misused constantly, no matter how capable the underlying LLM is.
Tool Definition Best Practices
from langchain_core.tools import tool
from pydantic import BaseModel, Field
class DatabaseQueryInput(BaseModel):
"""Input for querying the enterprise data warehouse."""
query: str = Field(
description="A read-only SQL query. Must include a LIMIT clause. "
"Available tables: clients, revenue, transactions. "
"Date columns use YYYY-MM-DD format."
)
database: str = Field(
default="analytics",
description="Target database: 'analytics' for historical data, "
"'realtime' for last-24h data."
)
@tool(args_schema=DatabaseQueryInput)
def query_database(query: str, database: str = "analytics") -> str:
"""Execute a read-only SQL query against the enterprise data warehouse.
Use this tool when you need to retrieve structured data like revenue
figures, client metrics, or transaction records. Always prefer this
over searching documents when the answer is in a database.
Returns: JSON string of query results (max 50 rows).
"""
# Validate query is read-only
if any(kw in query.upper() for kw in ["DROP", "DELETE", "UPDATE", "INSERT"]):
return "Error: Only SELECT queries are permitted."
# Execute with timeout and row limit
results = execute_with_timeout(query, database, timeout_seconds=30)
return format_results_as_json(results, max_rows=50)
The key principles from our production experience with LLM function calling:
- Descriptions are prompts. The tool description and field descriptions are essentially part of your prompt. Be specific about when to use the tool, what formats to expect, and what the constraints are.
- Fail loudly. Return clear error messages, not exceptions. The agent needs to understand what went wrong so it can retry intelligently.
- Constrain the output. Returning 10,000 rows will blow up your context window. Always cap results.
Code-Agents: Sandboxed Execution for Complex Analysis
One pattern that dramatically increased our agents' capability was giving them the ability to write and execute code. We call these "code-agents"—they generate Python for data analysis, statistical tests, or visualization, then execute it in a sandboxed environment.
@tool
def execute_python_analysis(code: str) -> str:
"""Run Python code in a sandboxed environment for data analysis.
The sandbox has access to: pandas, numpy, scipy, sklearn.
Data is pre-loaded as a DataFrame named 'df'.
Returns: stdout output and any generated file paths.
"""
import subprocess
import tempfile
import os
with tempfile.NamedTemporaryFile(
mode="w", suffix=".py", delete=False
) as f:
# Inject data loading preamble
preamble = "import pandas as pd\nimport numpy as np\n"
preamble += f"df = pd.read_parquet('/sandbox/data/current.parquet')\n"
f.write(preamble + code)
f.flush()
result = subprocess.run(
["python", f.name],
capture_output=True,
text=True,
timeout=60,
cwd="/sandbox",
env={**os.environ, "MPLBACKEND": "Agg"},
)
os.unlink(f.name)
if result.returncode != 0:
return f"Execution failed:\n{result.stderr[-1000:]}"
return result.stdout[-2000:]
In production, we use containerized sandboxes (gVisor on Kubernetes) rather than plain subprocess calls. The principle is the same: give the agent a powerful tool, but fence it in. At Presight AI, our code-agents handle roughly 30% of all analytical requests—anything involving statistical tests, time-series decomposition, or custom aggregations.
Multi-Agent Orchestration: Divide and Conquer
For complex enterprise workflows, a single agent is not enough. We use a supervisor pattern where a routing agent delegates to specialized sub-agents:
def create_supervisor():
"""Supervisor that routes to specialized agents."""
members = ["data_analyst", "report_writer", "search_agent"]
system_prompt = (
"You are a supervisor managing a team of specialized agents. "
"Based on the user's request, delegate to the appropriate agent. "
"For data queries, use data_analyst. "
"For document search, use search_agent. "
"For drafting summaries or reports, use report_writer. "
"You may call multiple agents in sequence."
)
class RouteDecision(BaseModel):
next: str = Field(
description="The next agent to call, or FINISH if done."
)
reasoning: str = Field(
description="Why this agent was chosen."
)
llm = ChatOpenAI(model="gpt-4o", temperature=0)
structured_llm = llm.with_structured_output(RouteDecision)
def supervisor_node(state: AgentState) -> AgentState:
messages = [SystemMessage(content=system_prompt)] + state["messages"]
decision = structured_llm.invoke(messages)
return {"next_action": decision.next, "messages": state["messages"]}
# Build the graph with sub-agent nodes
graph = StateGraph(AgentState)
graph.add_node("supervisor", supervisor_node)
for member in members:
graph.add_node(member, create_specialist_agent(member))
# Supervisor routes to specialists, specialists report back
graph.set_entry_point("supervisor")
graph.add_conditional_edges(
"supervisor",
lambda s: s["next_action"],
{m: m for m in members} | {"FINISH": END},
)
for member in members:
graph.add_edge(member, "supervisor")
return graph.compile()
The reasoning field in RouteDecision is not just for logging—it measurably improves routing accuracy. Forcing the LLM to articulate why it chose a particular agent acts as a chain-of-thought prompt that reduces misrouting by about 15% in our benchmarks.
Agentic Search: A Smarter Alternative to RAG
One of our most impactful discoveries was replacing our traditional vector-search RAG pipeline with an agentic search pattern. Instead of retrieving the top-k chunks and hoping for the best, our search agent iteratively refines its queries:
- Initial query decomposition: Break the user question into sub-queries
- Retrieve and evaluate: Fetch results, then use the LLM to judge relevance
- Refine or expand: If results are insufficient, reformulate the query
- Synthesize: Combine findings from multiple retrieval rounds
This approach increased our answer accuracy on complex research questions from 64% to 87%, because the agent can recognize when it has retrieved the wrong documents and course-correct—something a static RAG pipeline simply cannot do.
Evaluation: LLM-as-Judge in Production
You cannot improve what you cannot measure, and evaluating agentic systems is notoriously hard. We use a layered evaluation strategy:
Automated LLM-as-Judge Scoring
def evaluate_agent_response(query: str, response: str, ground_truth: str) -> dict:
"""Score agent response on multiple dimensions."""
judge_llm = ChatOpenAI(model="gpt-4o", temperature=0)
eval_prompt = f"""Score the following agent response on a scale of 1-5
for each dimension. Be strict—5 means perfect.
User Query: {query}
Agent Response: {response}
Reference Answer: {ground_truth}
Dimensions:
- correctness: Are the facts accurate?
- completeness: Does it fully answer the question?
- conciseness: Is it free of unnecessary information?
- tool_efficiency: Did the agent use the minimum necessary steps?
Return JSON with scores and brief justifications."""
result = judge_llm.invoke(eval_prompt)
return parse_eval_json(result.content)
Production Evaluation Pipeline
We run nightly evaluations against a curated test set of 200+ query-answer pairs, tracking scores over time. When we deploy a new model version (we self-host with vLLM for latency-sensitive workloads), the evaluation pipeline gates the rollout—if accuracy drops below our threshold, the deployment is automatically rolled back.
For rapid prototyping and non-critical workflows, we also use Dify as a low-code orchestration layer. It lets our domain experts (who are not engineers) build and iterate on agent workflows without touching code, which has been invaluable for expanding agentic AI adoption across the organization.
Lessons from Production
After running agentic systems serving thousands of enterprise users, here is what I wish someone had told me on day one:
- Start with one agent, not five. Multi-agent architectures are powerful but complex. Get a single agent working reliably before adding orchestration.
- Treat tool descriptions like documentation. Every misrouted tool call is a documentation failure, not a model failure.
- Log everything. Every LLM call, every tool invocation, every routing decision. You will need it for debugging and compliance.
- Set hard limits. Max iterations, max tokens per turn, max tool calls per session. Unbounded agents are expensive agents.
- Test with adversarial inputs. Users will ask ambiguous, contradictory, or impossible questions. Your agent should fail gracefully, not hallucinate.
Key Takeaways
- Agentic AI beats RAG for multi-step, tool-dependent enterprise workflows—but adds significant complexity. Choose the right tool for the job.
- LangGraph provides the state-machine control that enterprise deployments demand: auditability, deterministic routing, and explicit error handling.
- Tool design is 60% of the work. Invest in clear descriptions, input validation, and constrained outputs. Your agent is only as good as its tools.
- Code-agents with sandboxed execution unlock a class of analytical tasks that pure text agents cannot handle. Containerize and constrain them aggressively.
- Agentic search outperforms static RAG for complex queries by allowing iterative retrieval and self-correction.
- LLM-as-Judge evaluation is essential for continuous monitoring. Gate deployments on automated quality scores, and maintain a curated test set that grows over time.
- Production guardrails are not optional. Iteration limits, timeouts, read-only database access, and graceful failure modes are what separate a demo from a product.
The agentic AI landscape is evolving fast—new orchestration frameworks and model capabilities emerge every month. But the fundamentals in this guide will remain relevant: explicit state management, disciplined tool design, sandboxed execution, and rigorous evaluation. Build on those, and your enterprise agents will be ready for production.