HUYEN, Agents

Chip Huyen Intelligent agents are considered by many to be the ultimate goal of AI. The classic book by Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach (Prentice Hall, 1

We've completed the initial documentation for AI agents, drawing significantly from Chip Huyen's "AI Engineering" (2025). This document serves as a foundational overview, exploring the definition, components, capabilities, limitations, and evaluation of AI agents, with a particular focus on the practical implementation using foundation models. Here's a summary of the key areas covered: ### Agent Overview An agent is defined as anything that can perceive its environment and act upon that environment. Its capabilities are determined by the environment it operates in and the tools it has access to. We discussed how AI serves as the agent's "brain," processing tasks, planning action sequences, and determining task accomplishment. A key insight is that agents often require more powerful models due to the potential for compound mistakes across multiple steps and the higher stakes associated with their impactful tasks. ### Tools External tools vastly augment an agent's capabilities, enabling both perception (read-only actions) and interaction (write actions) with the environment. We categorized tools into three main types: * **Knowledge Augmentation:** Tools like text/image retrievers, SQL executors, and web browsing capabilities that provide up-to-date or specific information. We noted the importance of selecting internet APIs carefully to avoid exposing the agent to unreliable information. * **Capability Extension:** Tools that address inherent limitations of AI models, such as calculators for math, code interpreters for analysis and execution (with a warning about code injection risks), and multimodal tools (e.g., text-to-image models). * **Write Actions:** Tools that enable agents to make changes to data sources (e.g., updating a database, sending emails). We highlighted that while these enable greater automation, they also introduce significant security and trust concerns, emphasizing the need for robust security measures and human oversight. ### Agents and Security The discussion on security underscored the critical need for safety and trust when deploying autonomous AI agents, whether in physical or virtual environments. We recognized that while concerns are valid, advancements in security measures could eventually allow us to trust AI systems as much as or more than human counterparts in certain contexts. ### Planning Planning is central to an agent's ability to accomplish complex tasks. We explored the concept of planning as a search problem, involving understanding the task, considering options, and choosing the most promising path. Key aspects discussed include: * **Decoupling Planning from Execution:** To prevent wasteful execution, plans should ideally be validated (e.g., using heuristics or AI judges) before execution. This often leads to a multi-agent system where different components handle planning, validation, and execution. * **Human Involvement:** Humans can be integrated into any stage of the planning process (providing, validating, or executing parts of a plan) to aid, mitigate risks, or handle complex/risky operations. * **Reflection and Error Correction:** A robust agent workflow includes reflection steps to evaluate generated plans and execution outcomes, allowing for correction and iteration. ### Foundation Models as Planners We touched upon the ongoing debate about the inherent planning capabilities of foundation models, particularly autoregressive LLMs. While some argue against their ability to plan due to their forward-generating nature, others suggest that with appropriate tools and understanding of action outcomes, LLMs can form coherent plans. We recognize that an LLM can still be a crucial part of a planning system, even if it doesn't "plan" in the traditional sense, by being augmented with search and state-tracking tools. ### Foundation Model (FM) Versus Reinforcement Learning (RL) Planners We briefly compared FM agents and RL agents, noting their similarities in environment and actions, but differing in how their planners are trained (RL algorithms vs. prompting/fine-tuning FMs). We anticipate a future convergence of these two approaches. ### Plan Generation Prompt engineering is a straightforward way to turn a model into a plan generator. We illustrated with examples how to prompt a model to propose a sequence of actions, acknowledging that predicting exact function parameters can be challenging and prone to hallucination. Tips for improving agent planning include better system prompts, clearer tool descriptions, function refactoring, using stronger models, and fine-tuning. ### Function Calling This section elaborated on how model providers enable tool use, often termed "function calling." We described the general workflow: declaring tools with their execution entry points, parameters, and documentation, and then allowing the model to decide which tools to use and with what parameters. We emphasized the importance of inspecting parameter values to ensure correctness. ### Planning Granularity Finally, we discussed the concept of planning granularity, highlighting the tradeoff between detailed (harder to generate, easier to execute) and high-level (easier to generate, harder to execute) plans. Hierarchical planning was presented as a way to navigate this tradeoff. We also noted the challenges of hardcoding exact function names in plans due to evolving tool inventories, which could necessitate prompt updates or fine-tuning. This document will continue to evolve as the field of AI-powered agents progresses, incorporating new theoretical frameworks and best practices.

By Romain Peter