Learn to build large language model applications: vector databases, langchain, fine tuning and prompt engineering. Learn more


Huda Mahmood - Author
Huda Mahmood
| February 1

Retrieval augmented generation (RAG) has improved the function of large language models (LLM). It empowers generative AI to create more coherent and contextually relevant content. Let’s take a deeper look into understanding RAG.


What is retrieval augmented generation?


It is an AI framework and a type of natural language processing (NLP) model that enables the retrieval of information from an external knowledge base. It integrates retrieval-based and generation-based approaches to provide a robust database for LLMs.


A retrieval augmented generation model accesses a large pre-existing pool of knowledge to improve the quality of LLM-generated responses. It ensures that the information is more accurate and up-to-date by combining factual data with contextually relevant information.


By combining vector databases and LLM, the retrieval model has set up a standard for the search and navigation of data for generative AI. It has become one of the most used techniques for LLM.


retrieval augmented generation
An example illustrating retrieval augmentation – Source: LinkedIn


Benefits of RAG


While retrieval augmented generation improves LLM responses, it offers multiple benefits to the generative AI efforts of an organization.


Improved contextual awareness


The retrieval component allows access to a large knowledge base, enabling the model to generate contextually relevant information. Due to improved awareness of the context, the output generated is more coherent and appropriate.


Enhanced accuracy


An LLM using a retrieval model can produce accurate results with proper attribution, including citations of relevant sources. Access to a large and accurate database ensures that factually correct results are generated.


Adaptability to dynamic knowledge


The knowledge base of a retrieval model is regularly updated to ensure access to the latest information. The system integrates new information without retraining the entire program, ensuring quick adaptability. It enables the generative models to access the latest statistics and research.


Resource efficiency


Retrieval mechanisms enable the model to retrieve information from a large information base. The contextual relevance of the data enhances the accuracy of the results, making the process resource-efficient. It makes handling of large data volumes easier and makes the system cost-efficient.


Increased developer control


Developers use a retrieval augmented generation model to control the information base of a LLM. They can adapt the data to the changing needs of the user. Moreover, they can also restrict the accessibility of the knowledge base, giving them control of data authorization.


Large language model bootcamp


Frameworks for retrieval augmented generation


A RAG system combines a retrieval model with a generation model. Developers use frameworks and libraries available online to implement the required retrieval system. Let’s take a look at some of the common resources used for it.


Hugging face transformers


It is a popular library of pre-trained models for different tasks. It includes retrieval models like Dense Passage Retrieval (DPR) and generation models like GPT. The transformer allows the integration of these systems to generate a unified retrieval augmented generation model.


Facebook AI similarity search (FAISS)


FAISS is used for similarity search and clustering dense vectors. It plays a crucial role in building retrieval components of a system. Its use is preferred in models where vector similarity is crucial for the system.


PyTorch and TensorFlow


These are commonly used deep learning frameworks that offer immense flexibility in building RAG models. They enable the developers to create retrieval and generation models separately. Both models can then be integrated into a larger framework to develop a RAG model.




It is a Python framework that is built on Elasticsearch. It is suitable to build end-to-end conversational AI systems. The components of the framework are used for storage of information, retrieval models, and generation models.


Learn to build LLM applications


Use cases of RAG


Some common use cases and real-world applications are listed below.

Content creation


It primarily deals with writing articles and blogs. It is one of the most common uses of LLM where the retrieval models are used to generate coherent and relevant content. It can lead to personalized results for users that include real-time trends and relevant contextual information.


Real-time commentary


A retriever uses APIs to connect real-time information updates with an LLM. It is used to create a virtual commentator which can be integrated further to create text-to-speech models. IBM used this mechanism during the US Open 2023 for live commentary.


Question answering system


question answering through retrieval augmented generation
Question answering through retrieval augmented generation – Source: Medium


The ability of LLMs to generate contextually relevant content enables the retrieval model to function as a question-answering machine. It can retrieve factual information from an extensive knowledge base to create a comprehensive answer.


Language translation


Translation is a tricky process. A retrieval model can detect the context of phrases and words, enabling the generation of relevant translations. Access to external databases ensures the results are accurate and fluent for the users. The extensive information on available idioms and phrases in multiple languages ensures this use case of the retrieval model.


Educational assistance


The application of a retrieval model in the educational arena is an extension of question answering systems. It uses the said system, particularly for educational queries of users. In answering questions and generating academic content, the system can create more comprehensive results with contextually relevant information.


Future of RAG


The integration of retrieval and generation models in LLM is expected to grow in the future. The current trends indicate their increasing use in technological applications. Some common areas of future development of RAG include:


  • Improved architecture – the development of retrieval and generation models will result in the innovation of neural network architectures


  • Enhanced conversational agents – improved adaptation of knowledge base into retrieval model databases will result in more sophisticated conversational agents that can adapt to domain-specific information in an improved manner


  • Integration with multimodal information – including different types of information, including images and audio, can result in contextually rich responses that encompass a diverse range of media


  • Increased focus on ethical concerns – since data privacy and ethics are becoming increasingly important in today’s digital world, the retrieval models will also focus more on mitigating biases and ethical concerns from the development systems



Hence, retrieval augmented generation is an important aspect of large language models within the arena of generative AI. It has improved the overall content processing and promises an improved architecture of LLMs in the future.

Izma Aziz
Izma Aziz
| December 6

In this blog, we are enhancing our Language Model (LLM) experience by adopting the Retrieval-Augmented Generation (RAG) approach!

We’ll explore the fundamental architecture of RAG conceptually and delve deeper by implementing it through the Lang Chain orchestration framework and leveraging an open-source model from Hugging Face for both question answering and text embedding. 

So, let’s get started! 

Common hallucinations in large language models  

The most common problem faced by state-of-the-art LLMs is that they produce inaccurate or hallucinated responses. This mostly occurs when prompted with information not present in their training set, despite being trained on extensive data.


Large language model bootcamp


This discrepancy between the general knowledge embedded in the LLM’s weights and newer information can be bridged using RAG. The solution provided by RAG eliminates the need for computationally intensive and expertise-dependent fine-tuning, offering a more flexible approach to adapting to evolving information.


Read more about: AI hallucinations and risks associated with large language models




AI hallucinations
AI hallucinations

What is RAG? 

Retrieval-Augmented Generation involves enhancing the output of Large Language Models (LLMs) by providing them with additional information from an external knowledge source.



This method aims to improve the accuracy and contextuality of LLM-generated responses while minimizing factual inaccuracies. RAG empowers language models to sidestep the need for retraining, facilitating access to the most up-to-date information to produce trustworthy outputs through retrieval-based generation. 

Architecture of RAG approach

Retrieval augmented generation (RAG) - Elevate your large language models experience | Data Science Dojo

Figure from Lang chain documentation

Prerequisites for code implementation 

  1. HuggingFace account and LLAMA2 model access:
  • Create a Hugging Face account (free sign-up available) to access open-source Llama 2 and embedding models. 
  • Request access to LLAMA2 models using this form (access is typically granted within a few hours). 
  • After gaining access to Llama 2 models, please proceed to the provided link, select the checkbox to indicate your agreement to the information, and then click ‘Submit’. 

2. Google Colab account:

  • Create a Google account if you don’t already have one. 
  • Use Google Colab for code execution. 

3. Google Colab environment setup: 

  • In Google Colab, go to Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4 for faster execution of code. 

4. Library and dependency installation: 

  • Install necessary libraries and dependencies using the following command: 


5. Authentication with HuggingFace: 

  • Integrate your Hugging Face token into Colab’s environment:



  • When prompted, enter your Hugging Face token obtained from the “Access Token” tab in your Hugging Face settings. 


Step 1: Document Loading 

Loading a document refers to the process of retrieving and storing data as documents in memory from a specified source. This process is typically facilitated by document loaders, which provide a “load” method for accessing and loading documents into the memory. 

Lang chain has number of document loaders in this example we will be using “WebBaseLoader” class from the “langchain.document_loaders” module to load content from a specific web page.



The code extracts content from the web page “https://lilianweng.github.io/posts/2023-06-23-agent/“. BeautifulSoup (`bs4`) is employed for HTML parsing, focusing on elements with the classes “post-content”, “post-title”, and “post-header.” The loaded content is stored in the variable `docs`. 



Step 2: Document transformation – Splitting/chunking document 

After loading the data, it can be transformed to fit the application’s requirements or to extract relevant portions. This involves splitting lengthy documents into smaller chunks that are compatible with the model and produce accurate and clear results. Lang Chain offers various text splitters, in this implementation we chose the “RecursiveCharacterTextSplitter” for generic text processing.



The code breaks documents into chunks of 1000 characters with a 200-character overlap. This chunking is employed for embedding and vector storage, enabling more focused retrieval of relevant content during runtime. The recursive splitter ensures chunks maintain contextual integrity by using common separators, like new lines, until the desired chunk size is achieved. 

Step 3: Storage in vector database 

After extracting text chunks, we store and index them for future searches using the RAG application. A common approach involves embedding the content of each split and storing these embeddings in a vector store. 

When searching, we embed the search query and perform a similarity search to identify stored splits with embeddings most similar to the query embedding. Cosine similarity, which measures the angle between embeddings, is a simple similarity measure. 

Using the Chroma vector store and open source “HuggingFaceEmbeddings” in Lang chain, we can embed and store all document splits in a single command. 

Text embedding: 

Text embedding converts textual data into numerical vectors that capture the semantic meaning of the text. This enables efficient identification of similar text pieces. An embedding model, which is a variant of Language Models (LLMs) specifically designed for this purpose. 

 Lang Chain’s Embeddings class facilitates interaction with various text embedding models. While any model can be used, we opted for “HuggingFaceEmbeddings”. 




This code initializes an instance of the HuggingFaceEmbeddings class, configuring it with an open-source pre-trained model located at “sentence-transformers/all-MiniLM-l6-v2“. By doing this text embedding is created for converting textual data into numerical vectors. 


Learn to build custom large language model applications today!                                                


Vector Stores: 

Vector stores are specialized databases designed to efficiently store and search for high-dimensional vectors, such as text embeddings. They enable the retrieval of the most similar embedding vectors based on a given query vector. Lang Chain integrates with various vector stores, and we are using “Chroma” vector store for this task.



This code utilizes the Chroma class to create a vector store (vectorstore) from the previously split documents (splits) using the specified embeddings (embeddings). The Chroma vector store facilitates efficient storage and retrieval of document vectors for further processing. 

Step 4: Retrieval of text chunks 

After storing the data, preparing the LLM model, and constructing the pipeline, we need to retrieve the data. Retrievers serve as interfaces that return documents based on a query. 

Retrievers cannot store documents; they can only retrieve them. Vector stores form the foundation of retrievers. Lang Chain offers a variety of retriever algorithms, here is the one we implement. 



Step 5: Generation of answer with RAG approach 

Preparing the LLM Model: 

In the context of Retrieval Augmented Generation (RAG), an LLM model plays a crucial role in generating comprehensive and informative responses to user queries. By leveraging its ability to process and understand natural language, the LLM model can effectively combine retrieved documents with the given query to produce insightful and relevant outputs.


These lines import the necessary libraries for handling pre-trained models and tokenization. The specific model “meta-llama/Llama-2-7b-chat-hfis chosen for its question-answering capabilities.




This code defines a transformer pipeline, which encapsulates the pre-trained HuggingFace model and its associated configuration. It specifies the task as “text-generation” and sets various parameters to optimize the pipeline’s performance. 



This line creates a Lang Chain pipeline (HuggingFace Pipeline) that wraps the transformer pipeline. The model_kwargs parameter adjusts the model’s “temperature” to control its creativity and randomness. 

Retrieval QA Chain: 

To combine question-answering with a retrieval step, we employ the RetrievalQA chain, which utilizes a language model and a vector database as a retriever. By default, we process all data in a single batch and set the chain type to “stuff” when interacting with the language model. 






This code initializes a RetrievalQA instance by specifying a chain type (“stuff”), a HuggingFacePipeline (llm), and a retriever (retriever-initialize previously in the code from vectorstore). The return_source_documents parameter is set to True to include source documents in the output, enhancing contextual information retrieval.

Finally, we call this QA chain with the specific question we want to ask.



The result will be: 



We can print source documents to see which document chunks the model used to generate the answer to this specific query.





In this output, only 2 out of 4 document contents are shown as an example, that were retrieved to answer the specific question. 


In conclusion, by embracing the Retrieval-Augmented Generation (RAG) approach, we have elevated our Language Model (LLM) experience to new heights.

Through a deep dive into the conceptual foundations of RAG and practical implementation using the Lang Chain orchestration framework, coupled with the power of an open-source model from Hugging Face, we have enhanced question answering capabilities of LLMs.

This journey exemplifies the seamless integration of innovative technologies to optimize LLM capabilities, paving the way for a more efficient and powerful language processing experience. Cheers to the exciting possibilities that arise from combining innovative approaches with open-source resources! 

Muhammad Jan Author
Muhammad Jan
| October 31

RAG integration revolutionized search with LLM, boosting dynamic retrieval.

Within the implementation of a RAG system, a pivotal factor governing its efficiency and performance lies in the determination of the optimal chunk size. How does one identify the most effective chunk size for seamless and efficient retrieval? This is precisely where the comprehensive assessment provided by the LlamaIndex Response Evaluation tool becomes invaluable.

In this article, we will provide a comprehensive walkthrough, enabling you to discern the ideal chunk size through the powerful features of LlamaIndex’s Response Evaluation module. 


Why chunk size matters in RAG system

Selecting the appropriate chunk size is a crucial determination that holds sway over the effectiveness and precision of a RAG system in various ways: 




Pertinence and detail:

Opting for a smaller chunk size, such as 256, results in more detailed segments. However, this heightened detail brings the potential risk that pivotal information might not be included in the most retrieved segments.

On the contrary, a chunk size of 512 is likely to encompass all vital information within the leading chunks, ensuring that responses to inquiries are readily accessible. To navigate this challenge, we will employ the faithfulness and relevance metrics.

These metrics gauge the absence of ‘hallucinations’ and the ‘relevancy’ of responses concerning the query and the contexts retrieved, respectively. 


Large language model bootcamp

Generation time for responses:

With an increase in the chunk size, the volume of information directed into the LLM for generating a response also increases. While this can guarantee a more comprehensive context, it might potentially decelerate the system. Ensuring that the added depth doesn’t compromise the system’s responsiveness is pivotal.

Ultimately, finding the ideal chunk size boils down to achieving a delicate equilibrium. Capturing all crucial information while maintaining operational speed It’s essential to conduct comprehensive testing with different sizes to discover a setup that aligns with the unique use case and dataset requirements. 

Why evaluation? 

The discussion surrounding evaluation in the field of NLP has been contentious, particularly with the advancements in NLP methodologies.

Traditional evaluation techniques like BLEU or F1 are now unreliable for assessing models because they have limited correspondence with human evaluations.

As a result, the landscape of evaluation practices continues to shift, emphasizing the need for cautious application. 

In this blog, our focus will be on configuring the gpt-3.5-turbo model to serve as the central tool for evaluating the responses in our experiment.

To facilitate this, we establish two key evaluators, the faithfulness evaluator and the relevance evaluator, utilizing the service context. This approach aligns with the evolving standards of LLM evaluation, reflecting the need for more sophisticated and reliable evaluation mechanisms. 


 Faithfulness evaluator: This evaluator is instrumental in determining whether the response was artificially generated and checks if the response from a query engine corresponds with any source nodes. 

Relevancy evaluator: This evaluator is crucial for gauging whether the query was effectively addressed by the response and examines whether the response, combined with source nodes, matches the query. 

In order to determine the appropriate chunk size, we will calculate metrics such as average response time, average faithfulness, and average relevancy across different chunk sizes.  



Downloading dataset 

We will be using the IRS armed forces tax guide for this experiment. 

  • mkdir is used to make a folder. Here we are making a folder named dataset in the root directory. 
  • wget command is used for non-interactive downloading of files from the web. It allows users to retrieve content from web servers, supporting various protocols like HTTP, HTTPS, and FTP. 



Load dataset 

  • SimpleDirectoryReader class will help us to load all the files in the dataset directory. 
  • document[0:10] represents that we will only be loading the first 10 pages of the file for the sake of simplicity. 



Defining question bank 

These questions will help us to evaluate metrics for different chunk sizes. 




Establishing evaluators  

This code initializes an OpenAI language model (gpt-3.5-turbo) with temperature=0 settings and instantiate evaluators for measuring faithfulness and relevancy, utilizing the ServiceContext module with default configurations. 



Main evaluator method 

We will be evaluating each chunk size based on 3 metrics. 

  1. Average Response Time 
  2. Average Faithfulness 
  3. Average Relevancy 


Read this blog about Orchestation Framework


  • The function evaluator takes two parameters, chunkSize and questionBank. 
  • It first initializes an OpenAI language model (llm) with the model set to gpt-3.5-turbo. 
  • Then, it creates a serviceContext using the ServiceContext.from_defaults method, specifying the language model (llm) and the chunk size (chunkSize). 
  • The function uses the VectorStoreIndex.from_documents method to create a vector index from a set of documents, with the service context specified. 
  • It builds a query engine (queryEngine) from the vector index. 
  • The total number of questions in the question bank is determined and stored in the variable totalQuestions. 

Next, the function initializes variables for tracking various metrics: 

  • totalResponseTime: Tracks the cumulative response time for all questions. 
  • totalFaithfulness: Tracks the cumulative faithfulness score for all questions. 
  • totalRelevancy: Tracks the cumulative relevancy score for all questions. 
  • It records the start time before querying the queryEngine for a response to the current question. 
  • It calculates the elapsed time for the query by subtracting the start time from the current time. 
  • The function evaluates the faithfulness of the response using faithfulnessLLM.evaluate_response and stores the result in the faithfulnessResult variable. 
  • Similarly, it evaluates the relevancy of the response using relevancyLLM.evaluate_response and stores the result in the relevancyResult variable. 
  • The function accumulates the elapsed time, faithfulness result, and relevancy result in their respective total variables. 
  • After evaluating all the questions, the function computes the averages 




Testing different chunk sizes 

To find out the best chunk size for our data, we have defined a list of chunk sizes then we will traverse through the list of chunk sizes and find out the average response time, average faithfulness, and average relevance with the help of evaluator method. After this, we will convert our data list into a data frame with the help of Pandas DataFrame class to view it in a fine manner. 



From the illustration, it is evident that the chunk size of 128 exhibits the highest average faithfulness and relevancy while maintaining the second-lowest average response time. 

Use LlamaIndex to construct a RAG system 

Identifying the best chunk size for a RAG system depends on a combination of intuition and empirical data. By utilizing LlamaIndex’s Response Evaluation module, we can experiment with different sizes and make well-informed decisions.

When constructing a RAG system, it is crucial to remember that the chunk size plays a pivotal role. Therefore, it is essential to invest the necessary time to thoroughly evaluate and fine-tune the chunk size for optimal outcomes. 


You can find the complete code here