llm agents

Have you ever thought about the leap from “Good to Great” as James Collins describes in his book?

This is precisely what we aim to achieve with large language models (LLMs) today.

We are at a stage where language models are surely competent, but the challenge is to elevate them to excellence.

While there are numerous approaches that are being discussed currently to enhance LLMs, one approach that seems to be very promising is incorporating agentic workflows in LLMs.

Let’s dig deeper into what are AI agents, and how can they improve the results generated by LLMs.

What are Agentic Workflows

Agentic workflows are all about making LLMs smarter by integrating them into structured processes. This helps the AI deliver higher-quality results.

Right now, large language models usually operate on a zero-shot mode.

This equates to asking someone to write an 800-word blog on AI agents in one go, without any edits.

 

It’s not ideal, right?

 

That’s where AI agents come in. They let the LLM go over the task multiple times, fine-tuning the results each time. This process uses extra tools and smarter decision-making to really leverage what LLMs can do, especially for specific, targeted projects. Read more about AI agents

How AI Agents Enhance Large Language Models

Agent workflows have been proven to dramatically improve the performance of language models. For example, GPT 3.5 observed an increase in coding accuracy from 48.1% to 95.1% when moving from zero-shot prompting to an agent workflow on a coding benchmark.

Building Blocks for AI Agents

There is a lot of work going on globally about different strategies to create AI agents. To put the research into perspective, here’s a framework for categorizing design patterns for building agents.

 

1. Reflection

Reflection refers to a design pattern where an LLM generates an output and then reflects on its creation to identify improvement areas.

This process of self-critique allows the model to automatically provide constructive criticism of its output, much like a human would revise their work after writing a first draft.

Reflection leads to performance gains in AI agents by enabling them to self-criticize and improve through an iterative process.

When an LLM generates an initial output, it can be prompted to reflect on that output by checking for issues related to correctness, style, efficiency, and whatnot.

Reflection in Action

Here’s an example process of how Reflection leads to improved code:

1. Initially, an LLM receives a prompt to write code for a specific task, X.
2. Once the code is generated, the LLM reviews its work, assessing the code’s accuracy, style, and efficiency, and provides suggestions for improvements.
3. The LLM identifies any issues or opportunities for optimization and proposes adjustments based on this evaluation.
4. The LLM is prompted to refine the code, this time incorporating the insights gained from its own review.
5. This review and revision cycle continues, with the LLM providing ongoing feedback and making iterative enhancements to the code.

 

 

2. Tool Use

Incorporating different tools in the agenetic workflow allows the language model to call upon various tools for gathering information, taking actions, or manipulating data to accomplish tasks. This pattern extends the functionality of LLMs beyond generating text-based responses, allowing them to interact with external systems and perform more complex operations.

One can argue that some of the current consumer-facing products like ChatGPT are already capitalizing on different tools like web-search. Well, what we are proposing is different and massive. Here’s how:

• Access to Multiple Tools:

We are talking about AI Agents with the ability to access a variety of tools to perform a broad range of functions, from searching different sources (e.g., web, Wikipedia, arXiv) to interfacing with productivity tools (e.g., email, calendars).

This will allow LLMs to perform more complex tasks, such as managing communications, scheduling meetings, or conducting in-depth research—all in real-time.

Developers can use heuristics to include the most relevant subset of tools in the LLM’s context at each processing step, similar to how retrieval augmented generation (RAG) systems choose subsets of text for contextual relevance.

• Code Execution

One of the significant challenges with current LLMs is their limited ability to perform accurate computations directly from a trained model.

For instance, asking a typical LLM a math-related query like calculating compound interest might not yield the correct result.

This is where the integration of tools like Python into LLMs becomes invaluable. By allowing LLMs to execute Python code, they can precisely calculate and solve complex mathematical queries.

This capability not only enhances the functionality of LLMs in academic and professional settings but also boosts user trust in their ability to handle technical tasks effectively.

3. Multi-Agent Collaboration

Handling complex tasks can often be too challenging for a single AI agent, much like it would be for an individual person.

This is where multi-agent collaboration becomes crucial. By dividing these complex tasks into smaller, more manageable parts, each AI agent can focus on a specific segment where its expertise can be best utilized.

This approach mirrors how human teams operate, with different specialists taking on different roles within a project. Such collaboration allows for more efficient handling of intricate tasks, ensuring each part is managed by the most suitable agent, thus enhancing overall effectiveness and results.

How different AI agents can perform specialized roles within a single workflow?

In a multi-agent collaboration framework, various specialized agents work together within a single system to efficiently handle complex tasks. Here’s a straightforward breakdown of the process:

• Role Specialization: Each agent has a specific role based on its expertise. For example, a Product Manager agent might create a Product Requirement Document (PRD), while an Architect agent focuses on technical specifications.
• Task-Oriented Dialogue: The agents communicate through task-oriented dialogues, initiated by role-specific prompts, to effectively contribute to the project.
• Memory Stream: A memory stream records all past dialogues, helping agents reference previous interactions for more informed decisions, and maintaining continuity throughout the workflow.
• Self-Reflection and Feedback: Agents review their decisions and actions, using self-reflection and feedback mechanisms to refine their contributions and ensure alignment with the overall goals.
• Self-Improvement: Through active teamwork and learning from past projects, agents continuously improve, enhancing the system’s overall effectiveness.

This framework allows for streamlined and effective management of complex tasks by distributing them among specialized LLM agents, each handling aspects they are best suited for.

Such systems not only manage to optimize the execution of subtasks but also do so cost-effectively, scaling to various levels of complexity and broadening the scope of applications that LLMs can address.

Furthermore, the capacity for planning and tool use within the multi-agent framework enriches the solution space, fostering creativity and improved decision-making akin to a well-orchestrated team of specialists.

 

 

4. Planning

Planning is a design pattern that empowers large language models to autonomously devise a sequence of steps to achieve complex objectives.

Rather than relying on a single tool or action, planning allows an agent to dynamically determine the necessary steps to accomplish a task, which might not be pre-determined or decomposable into a set of subtasks in advance.

By decomposing a larger task into smaller, manageable subtasks, planning allows for a more systematic approach to problem-solving, leading to potentially higher-quality and more comprehensive outcomes

Impact of  Planning on Outcome Quality

The impact of Planning on outcome quality is multifaceted:

Adaptability: It gives AI agents the flexibility to adapt their strategies on the fly, making them capable of handling unexpected changes or errors in the workflow.
Dynamism: Planning allows agents to dynamically decide on the execution of tasks, which can result in creative and effective solutions to problems that are not immediately obvious.
Autonomy: It enables AI systems to work with minimal human intervention, enhancing efficiency and reducing the time to resolution.

Challenges of Planning

The use of Planning also presents several challenges:

• Predictability: The autonomous nature of Planning can lead to less predictable results, as the sequence of actions determined by the agent may not always align with human expectations.
• Complexity: As the complexity of tasks increases, so does the challenge for the LLM to predict precise plans. This necessitates further optimization of LLMs for task planning to handle a broader range of tasks effectively.

Despite these challenges, the field is rapidly evolving, and improvements in planning abilities are expected to enhance the quality of outcomes further while mitigating the associated challenges

 

 

The Future of Agentic Workflows in LLMs

 

Large language models (LLMs) have taken the world by storm with their ability to understand and generate human-like text. These AI marvels can analyze massive amounts of data, answer your questions in comprehensive detail, and even create different creative text formats, like poems, code, scripts, musical pieces, emails, letters, etc.

It’s like having a conversation with a computer that feels almost like talking to a real person!

However, LLMs on their own exist within a self-contained world of text. They can’t directly interact with external systems or perform actions in the real world. This is where LLM agents come in and play a transformative role.

 

LLM agents act as powerful intermediaries, bridging the gap between the LLM’s internal world and the vast external world of data and applications. They essentially empower LLMs to become more versatile and take action on their behalf. Think of an LLM agent as a personal assistant for your LLM, fetching information and completing tasks based on your instructions.

For instance, you might ask an LLM, “What are the next available flights to New York from Toronto?” The LLM can access and process information but cannot directly search the web – it is reliant on its training data.

An LLM agent can step in, retrieve the data from a website, and provide the available list of flights to the LLM. The LLM can then present you with the answer in a clear and concise way.

 

 

By combining LLMs with agents, we unlock a new level of capability and versatility. In the following sections, we’ll dive deeper into the benefits of using LLM agents and explore how they are revolutionizing various applications.

Benefits and Use-cases of LLM Agents

Let’s explore in detail the transformative benefits of LLM agents and how they empower LLMs to become even more powerful.

Enhanced Functionality: Beyond Text Processing

LLMs excel at understanding and manipulating text, but they lack the ability to directly access and interact with external systems. An LLM agent bridges this gap by allowing the LLM to leverage external tools and data sources.

Imagine you ask an LLM, “What is the weather forecast for Seattle this weekend?” The LLM can understand the question but cannot directly access weather data. An LLM agent can step in, retrieve the forecast from a weather API, and provide the LLM with the information it needs to respond accurately.

This empowers LLMs to perform tasks that were previously impossible, like: 

• Accessing and processing data from databases and APIs 
• Executing code 
• Interacting with web services 

Increased Versatility: A Wider Range of Applications

By unlocking the ability to interact with the external world, LLM agents significantly expand the range of applications for LLMs. Here are just a few examples: 

• Data Analysis and Processing: LLMs can be used to analyze data from various sources, such as financial reports, social media posts, and scientific papers. LLM agents can help them extract key insights, identify trends, and answer complex questions. 
• Content Generation and Automation: LLMs can be empowered to create different kinds of content, like articles, social media posts, or marketing copy. LLM agents can assist them by searching for relevant information, gathering data, and ensuring factual accuracy. 
• Custom Tools and Applications: Developers can leverage LLM agents to build custom tools that combine the power of LLMs with external functionalities. Imagine a tool that allows an LLM to write and execute Python code, search for information online, and generate creative text formats based on user input. 

 

Explore the dynamics and working of agents in LLM

 

Improved Performance: Context and Information for Better Answers

LLM agents don’t just expand what LLMs can do, they also improve how they do it. By providing LLMs with access to relevant context and information, LLM agents can significantly enhance the quality of their responses: 

• More Accurate Responses: When an LLM agent retrieves data from external sources, the LLM can generate more accurate and informative answers to user queries. 
• Enhanced Reasoning: LLM agents can facilitate a back-and-forth exchange between the LLM and external systems, allowing the LLM to reason through problems and arrive at well-supported conclusions. 
• Reduced Bias: By incorporating information from diverse sources, LLM agents can mitigate potential biases present in the LLM’s training data, leading to fairer and more objective responses. 

Enhanced Efficiency: Automating Tasks and Saving Time

LLM agents can automate repetitive tasks that would otherwise require human intervention. This frees up human experts to focus on more complex problems and strategic initiatives. Here are some examples: 

• Data Extraction and Summarization: LLM agents can automatically extract relevant data from documents and reports, saving users time and effort. 
• Research and Information Gathering: LLM agents can be used to search for information online, compile relevant data points, and present them to the LLM for analysis. 
• Content Creation Workflows: LLM agents can streamline content creation workflows by automating tasks like data gathering, formatting, and initial drafts. 

In conclusion, LLM agents are a game-changer, transforming LLMs from powerful text processors to versatile tools that can interact with the real world. By unlocking enhanced functionality, increased versatility, improved performance, and enhanced efficiency, LLM agents pave the way for a new wave of innovative applications across various domains.

In the next section, we’ll explore how LangChain, a framework for building LLM applications, can be used to implement LLM agents and unlock their full potential.

 

 

Implementing LLM Agents with LangChain 

Now, let’s explore how LangChain, a framework specifically designed for building LLM applications, empowers us to implement LLM agents. 

What is LangChain?

LangChain is a powerful toolkit that simplifies the process of building and deploying LLM applications. It provides a structured environment where you can connect your LLM with various tools and functionalities, enabling it to perform actions beyond basic text processing. Think of LangChain as a Lego set for building intelligent applications powered by LLMs.

 

 

Implementing LLM Agents with LangChain: A Step-by-Step Guide

Let’s break down the process of implementing LLM agents with LangChain into manageable steps: 

Setting Up the Base LLM

The foundation of your LLM agent is the LLM itself. You can either choose an open-source model like Llama2 or Mixtral, or a proprietary model like OpenAI’s GPT or Cohere. 

Defining the Tools

Identify the external functionalities your LLM agent will need. These tools could be: 

• APIs: Services that provide programmatic access to data or functionalities (e.g., weather API, stock market API) 
• Databases: Collections of structured data your LLM can access and query (e.g., customer database, product database) 
• Web Search Tools: Tools that allow your LLM to search the web for relevant information (e.g., duckduckgo, serper API) 
• Coding Tools: Tools that allow your LLM to write and execute actual code (e.g., Python REPL Tool)

 

 

You can check out LangChain’s documentation to find a comprehensive list of tools and toolkits provided by LangChain that you can easily integrate into your agent, or you can easily define your own custom tool such as a calculator tool.

Creating an Agent

This is the brain of your LLM agent, responsible for communication and coordination. The agent understands the user’s needs, selects the appropriate tool based on the task, and interprets the retrieved information for response generation. 

Defining the Interaction Flow

Establish a clear sequence for how the LLM, agent, and tools interact. This flow typically involves: 

• Receiving a user query 
• The agent analyzes the query and identifies the necessary tools 
• The agent passes in the relevant parameters to the chosen tool(s) 
• The LLM processes the retrieved information from the tools
• The agent formulates a response based on the retrieved information 

Integration with LangChain

LangChain provides the platform for connecting all the components. You’ll integrate your LLM and chosen tools within LangChain, creating an agent that can interact with the external environment. 

Testing and Refining

Once everything is set up, it’s time to test your LLM agent! Put it through various scenarios to ensure it functions as expected. Based on the results, refine the agent’s logic and interactions to improve its accuracy and performance. 

By following these steps and leveraging LangChain’s capabilities, you can build versatile LLM agents that unlock the true potential of LLMs.

 

 

LangChain Implementation of an LLM Agent with tools

In the next section, we’ll delve into a practical example, walking you through a Python Notebook that implements a LangChain-based LLM agent with retrieval (RAG) and web search tools. OpenAI’s GPT-4 has been used as the LLM of choice here. This will provide you with a hands-on understanding of the concepts discussed here. 

The agent has been equipped with two tools: 

1. A retrieval tool that can be used to fetch information from a vector store of Data Science Dojo blogs on the topic of RAG. LangChain’s PyPDFLoader is used to load and chunk the PDF blog text, OpenAI embeddings are used to embed the chunks of data, and Weaviate client is used for indexing and storage of data. 

1. A web search tool that can be used to query the web and bring up-to-date and relevant search results based on the user’s question. Google Serper API is used here as the search wrapper – you can also use duckduckgo search or Tavily API. 

Below is a diagram depicting the agent flow:

 

 

Let’s now start going through the code step-by-step. 

Installing Libraries

Let’s start by downloading all the necessary libraries that we’ll need. This includes libraries for handling language models, API clients, and document processing.

 

Importing and Setting API Keys

Now, we’ll ensure our environment has access to the necessary API keys for OpenAI and Serper by importing them and setting them as environment variables. 

 

Documents Preprocessing: Mounting Google Drive and Loading Documents

Let’s connect to Google Drive and load the relevant documents. I‘ve stored PDFs of various Data Science Dojo blogs related to RAG, which we’ll use for our tool. Following are the links to the blogs I have used: 

1. https://datasciencedojo.com/blog/rag-with-llamaindex/ 

1. https://datasciencedojo.com/blog/llm-with-rag-approach/ 

1. https://datasciencedojo.com/blog/efficient-database-optimization/ 

1. https://datasciencedojo.com/blog/rag-llm-and-finetuning-a-guide/ 

1. https://datasciencedojo.com/blog/rag-vs-finetuning-llm-debate/ 

1. https://datasciencedojo.com/blog/challenges-in-rag-based-llm-applications/ 

 

Extracting Text from PDFs

Using the PyPDFLoader from Langchain, we’ll extract text from each PDF by breaking them down into individual pages. This helps in processing and indexing them separately. 

 

Embedding and Indexing through Weaviate: Embedding Text Chunks

Now we’ll use Weaviate client to turn our text chunks into embeddings using OpenAI’s embedding model. This prepares our text for efficient querying and retrieval.

 

Setting Up the Retriever

With our documents embedded, let’s set up the retriever which will be crucial for fetching relevant information based on user queries.

 

Defining Tools: Retrieval and Search Tools Setup

Next, we define two key tools: one for retrieving information from our indexed blogs, and another for performing web searches for queries that extend beyond our local data.

 

Adding Tools to the List

We then add both tools to our tool list, ensuring our agent can access these during its operations.

 

Setting up the Agent: Creating the Prompt Template

Let’s create a prompt template that guides our agent on how to handle different types of queries using the tools we’ve set up. 

 

Initializing the LLM with GPT-4

For the best performance, I used GPT-4 as the LLM of choice as GPT-3.5 seemed to struggle with routing to tools correctly and would go back and forth between the two tools needlessly.

 

Creating and Configuring the Agent

With the tools and prompt template ready, let’s construct the agent. This agent will use our predefined LLM and tools to handle user queries.

 

 

Invoking the Agent: Agent Response to a RAG-related Query

Let’s put our agent to the test by asking a question about RAG and observing how it uses the tools to generate an answer.

 

Agent Response to an Unrelated Query

Now, let’s see how our agent handles a question that’s not about RAG. This will demonstrate the utility of our web search tool.

 

 

That’s all for the implementation of an LLM Agent through LangChain. You can find the full code here.

 

 

This is, of course, a very basic use case but it is a starting point. There is a myriad of stuff you can do using agents and LangChain has several cookbooks that you can check out. The best way to get acquainted with any technology is to actually get your hands dirty and use the technology in some way.

I’d encourage you to look up further tutorials and notebooks using agents and try building something yourself. Why not try delegating a task to an agent that you yourself find irksome – perhaps an agent can take off its burden from your shoulders!

LLM agents: A building block for LLM applications

To sum it up, LLM agents are a crucial element for building LLM applications. As you navigate through the process, make sure to consider the role and assistance they have to offer.