until LLM Bootcamp: In-Person (Seattle) and Online Learn more

Llama 2

7B refers to a specific model size for large language models (LLMs) consisting of seven billion parameters. With the growing importance of LLMs, there are several options in the market. Each option has a particular model size, providing a wide range of choices to users.

However, in this blog we will explore two LLMs of 7B – Mistral 7B and Llama-2 7B, navigating the differences and similarities between the two options. Before we dig deeper into the showdown of the two 7B LLMs, let’s do a quick recap of the language models.


Large language model bootcamp


Understanding Mistral 7B and Llama-2 7B

Mistral 7B is an LLM powerhouse created by Mistral AI. The model focuses on providing enhanced performance and increased efficiency with reduced computing resource utilization. Thus, it is a useful option for conditions where computational power is limited.

Moreover, the Mistral LLM is a versatile language model, excelling at tasks like reasoning, comprehension, tackling STEM problems, and even coding.


Read more and gain deeper insight into Mistral 7B


On the other hand, Llama-2 7B is produced by Meta AI to specifically target the art of conversation. The researchers have fine-tuned the model, making it a master of dialog applications, and empowering it to generate interactive responses while understanding the basics of human language.

The Llama model is available on platforms like Hugging Face, allowing you to experiment with it as you navigate the conversational abilities of the LLM. Hence, these are the two LLMs with the same model size that we can now compare across multiple aspects.

Battle of the 7Bs: Mistral vs Llama

Now, we can take a closer look at comparing the two language models to understand the aspects of their differences.


When it comes to performance, Mistral AI’s model excels in its ability to handle different tasks. It has successfully reached the benchmark scores with every standardized test for various challenges in reasoning, comprehension, problem-solving, and much more.

On the contrary, Meta AI‘s production takes on a specialized approach. In this case, the art of conversation. While it will not score outstanding results and produce benchmark scores for a variety of tasks, its strength lies in its ability to understand and respond fluently within a dialogue.


A visual comparison of the performance parameters of the 7Bs
A visual comparison of the performance parameters of the 7Bs – Source: E2E Cloud



Mistral 7B operates with remarkable efficiency due to the adoption of a technique called Group-Query Attention (GQA). It allows the language model to group similar queries for faster inference and results.

GQA is the middle ground between the quality of Multi-Head Attention (MHA) and the speed of Multi-Query Attention (MQA) approaches. Hence, allowing the model to strike a balance between performance and efficiency.

However, scarce knowledge of the training data of Llama-2 7B limits the understanding of its efficiency. We can still say that a broader and more diverse dataset can enhance the model’s efficiency in producing more contextually relevant responses.


When it comes to accessibility of the two models, both are open-source resources that are open for use and experimentation. It can be noted though, that the Llama-2 model offers easier access through platforms like Hugging Face.

Meanwhile, the Mistral language model requires some deeper navigation and understanding of the resources provided by Mistral AI. It demands some research, unlike its competitor for information access.

Hence, these are some notable differences between the two language models. While these aspects might determine the usability and access of the models, each one has the potential to contribute to the development of LLM applications significantly.


How generative AI and LLMs work


Choosing the right model

Since we understand the basic differences, the debate comes down to selecting the right model for use. Based on the highlighted factors of comparison here, we can say that Mistral is an appropriate choice for applications that require overall efficiency and high performance in a diverse range of tasks.

Meanwhile, Llama-2 is more suited for applications that are designed to attain conversational prowess and dialog expertise. While this distinction of use makes it easier to pick the right model, some key factors to consider also include:

  • Future Development – Since both models are new, you must stay in touch with their ongoing research and updates. These advancements can bring new information to light, impacting your model selection.
  • Community Support – It is a crucial factor for any open-source tool. Investigate communities for both models to get a better understanding of the models’ power. A more active and thriving community will provide you with valuable insights and assistance, making your choice easier.



Future prospects for the language models

As the digital world continues to evolve, it is accurate to expect the language models to update into more powerful resources in the future. Among some potential routes for Mistral 7B is the improvement of GQA for better efficiency and the ability to run on even less powerful devices.

Moreover, Mistral AI can make the model more readily available by providing access to it through different platforms like Hugging Face. It will also allow a diverse developer community to form around it, opening doors for more experimentation with the model.


Explore a hands-on curriculum that helps you build custom LLM applications!


As for Llama-2 7B, future prospects can include advancements in dialog modeling. Researchers can work to empower the model to understand and process emotions in a conversation. It can also target multimodal data handling, going beyond textual inputs to handle audio or visual inputs as well.

Thus, we can speculate several trajectories for the development of these two language models. In this discussion, it can be said that no matter in what direction, an advancement of the models is guaranteed in the future. It will continue to open doors for improved research avenues and LLM applications.

April 23, 2024

In this blog, we will be getting started with the Llama 2 open-source large language model. We will guide you through various methods of accessing it, ensuring that by the end, you will be well-equipped to unlock the power of this remarkable language model for your projects.

Whether you are a developer, researcher, or simply curious about its capabilities, this blog will equip you with the knowledge and tools you need to get started. 


Understanding Llama 2 

In the ever-evolving landscape of artificial intelligence, language models have emerged as pivotal tools for developers, researchers, and enthusiasts alike. One such remarkable addition to the world of language models is Llama 2. While it may not be the absolute marvel of language models, it stands out as an open-source gem. 

Llama 2, an open-source large language model, opens its doors for both research and commercial use, breaking down barriers to innovation and creativity. It comprises a range of pre-trained and fine-tuned generative text models, varying in scale from 7 billion to a staggering 70 billion parameters.


Read more about – > Llama 2 fine-tuning


Among these, the Llama-2-Chat models, optimized for dialogue, shine as they outperform open-source chat models across various benchmarks. In fact, their helpfulness and safety evaluations rival some popular closed-source models like ChatGPT and PaLM. 

In this blog, we will exploring its training process, improvements over its predecessor, and ways to harness its potential.



If you want to use it in your projects, this guide will get you started.

So, let us embark on this journey together as we unveil the world of Llama 2 and discover how it can elevate your AI (Artificial Intelligence) endeavors. 


Llama 2: The evolution and enhanced features 


It represents a significant leap forward from its predecessor, Llama 1, which garnered immense attention and demand from researchers worldwide. With over 100,000 requests for access, the research community demonstrated its appetite for powerful language models.

Building upon this foundation, Llama 2 emerges as the next generation offering from Meta, succeeding its predecessor, Llama 1. Unlike Llama 1, which was released under a non-commercial license for research purposes, it takes a giant stride by making itself available freely for both research and commercial applications. 

 Large language model bootcamp

This second-generation model comes with notable enhancements, including pre-trained versions with parameter sizes of 7 billion, 13 billion, and a staggering 70 billion. Llama 2’s training data has been expanded, encompassing 40% more information, all while boasting double the context length compared to Llama 1, with a context length of 4,096 tokens.


Notably, the Llama-2 chat models, tailored for dialogue applications, have been fine-tuned with the assistance of over 1 million new human annotations. As we delve deeper, we will explore its capabilities and the numerous ways to access this remarkable language model. 

Source: https://ai.meta.com/llama/ 


Exploring your path to Llama 2: Six access methods you must learn 

Accessing the power of it is easier than you might think, thanks to its open-source nature. Whether you are a researcher, developer, or simply curious, here are six ways to get your hands on the Llama 2 model right now: 


Unlocking_Llama2_six access methods
Understanding Llama2, Six Access Methods



Download Llama 2 Model 

Since Llama 2 large language model is open-source, you can freely install it on your desktop and start using it. For this, you will need to complete a few simple steps. 

  • First, head to Meta AI’s official Llama 2 download webpage and fill in the requested information. Make sure you select the right model you plan on utilizing. 
Llama2 Download Request Form


  • Upon submitting your download request, you can expect to encounter the following page. You will receive an installation email from Meta with more information regarding the download. 




  • Once the email has been received, you can proceed with the installation by adhering to the instructions detailed within the email. To begin, the initial step entails accessing the Llama repository on GitHub.  


  • Download the code and extract the ZIP file to your desktop. Subsequently, proceed by adhering to the instructions outlined in the “Readme” document to start using all available models. 


Learn to build custom large language model applications today!                                                


Its models are also available in the Hugging Face organization of Llama 2 from Meta. All the available models are accessible there as well. To use these models from Hugging Face, we still need to submit a download request to Meta, and additionally, we need to fill out a form to enable the use of Llama 2 in Hugging Face.

To access its models on Hugging Face, follow these steps:


Meta Llama2 Organization HuggingFace


  • You can see a “Models” tab on the page which lists all the available models. 


Access Llama2 HuggingFace



  • In the Access Llama 2 on Hugging Face card enter the email you used to send out the download request. 

Note: Please ensure that the email you use on Hugging Face matches the one you used to request Llama 2 download permission from Meta. 


Utilize the quantized model from Hugging Face 

In addition to the models from the official Meta Llama 2 organization, there are some quantized models also available on Hugging Face. 

If you search for Llama in the Hugging Face search bar. You will see a list of models available in Hugging Face. You can see that models from meta-llama the official organization are available but there are other models also available.

These models are the quantized version of the same Llama 2 models. Like the model, TheBloke/Llama-2-7b-Chat-GGUF contains GGUF format model files for Meta Llama 2’s Llama 2 7B Chat. 





The key advantage of these compressed models lies in their accessibility. They are open-source and do not necessitate users to request downloads from either Meta or Hugging Face. Although they are not the complete, original models, these quantized versions allow users to harness the capabilities of the model with reduced computational requirements. 




Deploy Llama 2 on Microsoft Azure 

Microsoft and Meta have strengthened their partnership, designating Microsoft as the preferred partner for Llama 2. This collaboration brings Llama 2 into the Azure AI model catalog, granting developers using Microsoft Azure the capability to seamlessly integrate and utilize this powerful language model. 


Azure ML Model Catalog
Azure ML Model Catalog


Within the Azure model catalog, you can effortlessly locate the Llama 2 model developed by Meta. Microsoft Azure simplifies the fine-tuning of Llama 2, offering both UI-based and code-based methods to customize the model according to your requirements. Furthermore, you can assess the model’s performance with your test data to ascertain its suitability for your unique use case. 


Harness Llama 2 as a cloud-based API 

Another avenue to tap into the capabilities of the Llama 2 model is through the deployment of Llama 2 models on platforms such as Hugging Face and Replicate, transforming it into a cloud API. By leveraging the Hugging Face Inference Endpoint, you can establish an accessible endpoint for your Llama 2 model hosted on Hugging Face, facilitating its utilization. 

Hugging Face Inference Endpoint
Hugging Face Inference Endpoint


Additionally, it is conveniently accessible through Replicate, presenting a streamlined method for deploying and employing the model via API. This approach alleviates worries about the availability of GPU computing power, whether in the context of development or testing.

It enables the fine-tuning and operation of models in a cloud environment, eliminating the need for dedicated GPU setups. Serving as a cloud API, it simplifies the integration process for applications developed on a wide range of technologies. 



Online Interactions with Llama 2 

Experience its capabilities online through platforms like llama2.ai where you can freely engage with different models. Customize your interactions by adjusting parameters such as system prompt, max token, and randomness, offering a user-friendly gateway to explore the model’s creative AI potential.

This demo provides a non-technical audience with the opportunity to submit queries and toggle between chat modes, simplifying the experience of interacting with Llama 2’s generative abilities.  



Offline Llama 2 Interaction with LM Studio 

With LM Studio, you have the power to run LLMs (Large Language Models) offline on your laptop, employ models through an intuitive in-app Chat UI or compatible local servers, access model files from Hugging Face repositories, and discover exciting new LLMs right from the app’s homepage. 



LM Studio empowers you to engage with Llama 2 models offline. Here is how it works:  

  • Once installed, search for your desired Llama 2 model, such as Llama 2 7b. You will find a comprehensive list of repositories and quantized models on Hugging Face. Select your preferred repository and initiate the model download by clicking the link on the right. Monitor the download progress at the bottom of the screen. 




  • After the model is downloaded, click the AI Chat icon, select your model, and start a conversation with it. LM Studio offers a seamless offline experience, enabling you to explore the potential of Llama 2 models with ease. 
LM Studio Llama2 Inference
LM Studio Llama2 Inference


Explore Llama 2 now!

In summary, this blog has guided you on an exploration of an open-source language model.

We analyzed its development, pointed out its unique features, and gave a detailed overview of six methods to use it. These methods are suitable for developers, researchers, and anyone interested in their potential.

Armed with this understanding, you are now well-equipped to unlock the capabilities of Llama 2 for your individual AI initiatives and pursuits. 

October 25, 2023

With the introduction of LLaMA v1, we witnessed a surge in customized models like Alpaca, Vicuna, and WizardLM. This surge motivated various businesses to launch their own foundational models, such as OpenLLaMA, Falcon, and XGen, with licenses suitable for commercial purposes. LLaMA 2, the latest release, now combines the strengths of both approaches, offering an efficient foundational model with a more permissive license. 


In the first half of 2023, the software landscape underwent a significant transformation with the widespread adoption of APIs like OpenAI API to build infrastructures based on Large Language Models (LLMs). Libraries like LangChain and LlamaIndex played crucial roles in this evolution.  

Large language model bootcamp



As we move into the latter part of the year, fine-tuning or instruction tuning of these models is becoming a standard practice in the LLMOps workflow. This trend is motivated by several factors, including  


  • Potential cost savings 
  • The capacity to handle sensitive data 
  • The opportunity to develop models that can outperform well-known models like ChatGPT and GPT-4 in specific tasks. 



Fine-tuning methods refer to various techniques used to enhance the performance of a pre-trained model by adapting it to a specific task or domain. These methods are valuable for optimizing a model’s weights and parameters to excel in the target task. Here are different fine-tuning methods: 




  • Supervised Fine-Tuning: This method involves further training a pre-trained language model (LLM) on a specific downstream task using labeled data. The model’s parameters are updated to excel in this task, such as text classification, named entity recognition, or sentiment analysis. 


  • Transfer Learning: Transfer learning involves repurposing a pre-trained model’s architecture and weights for a new task or domain. Typically, the model is initially trained on a broad dataset and is then fine-tuned to adapt to specific tasks or domains, making it an efficient approach. 


  • Sequential Fine-tuning: Sequential fine-tuning entails the gradual adaptation of a pre-trained model on multiple related tasks or domains in succession. This sequential learning helps the model capture intricate language patterns across various tasks, leading to improved generalization and performance. 


  • Task-specific Fine-tuning: Task-specific fine-tuning is a method where the pre-trained model undergoes further training on a dedicated dataset for a particular task or domain. While it demands more data and time than transfer learning, it can yield higher performance tailored to the specific task. 


  • Multi-task Learning: Multi-task learning involves fine-tuning the pre-trained model on several tasks simultaneously. This strategy enables the model to learn and leverage common features and representations across different tasks, ultimately enhancing its ability to generalize and perform well. 


  • Adapter Training: Adapter training entails training lightweight modules that are integrated into the pre-trained model. These adapters allow for fine-tuning on specific tasks without interfering with the original model’s performance on other tasks. This approach maintains efficiency while adapting to task-specific requirements. 


Why fine-tune LLM? 


Fine tuning LLM

Source: DeepLearningAI 


The figure discusses the allocation of AI tasks within organizations, taking into account the amount of available data. On the left side of the spectrum, having a substantial amount of data allows organizations to train their own models from scratch, albeit at a high cost.

Alternatively, if an organization possesses a moderate amount of data, it can fine-tune pre-existing models to achieve excellent performance. For those with limited data, the recommended approach is in-context learning, specifically through techniques like retrieval augmented generation using general models.

However, our focus will be on the fine-tuning aspect, as it offers a favorable balance between accuracy, performance, and speed compared to larger, more general models. 


Pre-trained LLM

Source: Intuitive Tutorials 


Why LLaMA 2? 

Before we dive into the detailed guide, let’s take a quick look at the benefits of Llama 2. 

 Read more about Palm 2 vs Llama 2 in this blog


  • Diverse range: Llama 2 comes in various sizes, from 7 billion to a massive 70 billion parameters. It shares a similar architecture with Llama 1 but boasts improved capabilities.
  • Extensive training ata: This model has been trained on a massive dataset of 2 trillion tokens, demonstrating its vast exposure to a wide range of information. 
  • Enhanced context: With an extended context length of 4,000 tokens, the model can better understand and generate extensive content. 
  • Grouped query attention (GQA): GQA has been introduced to enhance inference scalability, making attention calculations faster by storing previous token pair information. 
  • Performance excellence: Llama 2 models consistently outperform their predecessors, particularly the Llama 2 70B version. They excel in various benchmarks, competing strongly with models like Llama 1 65B and even Falcon models. 
  •  Open source vs. closed source LLMs: When compared to models like GPT-3.5 or PaLM (540B), Llama 2 70B demonstrates impressive performance. While there may be a slight gap in certain benchmarks when compared to GPT-4 and PaLM-2, the model’s potential is evident. 

Parameter efficient fine-tuning (PEFT) 

Parameter Efficient Fine-Tuning involves adapting pre-trained models to new tasks while making minimal changes to the model’s parameters. This is especially important for large neural network models like BERT, GPT, and similar ones. Let’s delve into why PEFT is so significant:


  • Reduced overfitting: Limited datasets can be problematic. Making too many parameter adjustments can lead to model overfitting. PEFT allows us to strike a balance between the model’s flexibility and tailoring it to new tasks. 
  • Faster training: Making fewer parameter changes results in fewer computations, which in turn leads to faster training sessions. 
  • Resource efficiency: Training deep neural networks requires substantial computational resources. PEFT minimizes the computational and memory demands, making it more practical to deploy in resource-constrained environments.  
  • Knowledge preservation: Extensive pretraining on diverse datasets equips models with valuable general knowledge. PEFT ensures that this wealth of knowledge is retained when adapting the model to new tasks. 

Learn to build LLM applications

PEFT technique 

The most popular PEFT technique is LoRA. Let’s see what it offers: 


  • LoRA 

LoRA, or Low Rank Adaptation, represents a groundbreaking advancement in the realm of large language models. At the beginning of the year, these models seemed accessible only to wealthy companies. However, LoRA has changed the landscape. 

LoRA has made the use of large language models accessible to a wider audience. Its low-rank adaptation approach has significantly reduced the number of trainable parameters by up to 10,000 times. This results in:  

  • A threefold reduction in GPU requirements, which is typically a major bottleneck. 
  • Comparable, if not superior, performance even without fine-tuning the entire model. 

In traditional fine-tuning, we modify the existing weights of a pre-trained model using new examples. Conventionally, this required a matrix of the same size. However, by employing creative methods and the concept of rank factorization, a matrix can be split into two smaller matrices. When multiplied together, they approximate the original matrix. 

To illustrate, imagine a 1000×1000 matrix with 1,000,000 parameters. Through rank factorization, if the rank is, for instance, five, we could have two matrices, each sized 1000×5. When combined, they represent just 10,000 parameters, resulting in a significant reduction. 

In recent days, researchers have introduced an extension of LoRA known as QLoRA. 

  • QLoRA 

QLoRA is an extension of LoRA that further introduces quantization to enhance parameter efficiency during fine-tuning. It builds on the principles of LoRA while introducing 4-bit NormalFloat (NF4) quantization and Double Quantization techniques. 


Quantization + LoRA

Environment setup 

About dataset 

 The dataset has undergone special processing to ensure a seamless match with Llama 2’s prompt format, making it ready for training without the need for additional modifications. 



Since the data has already been adapted to Llama 2’s prompt format, it can be directly employed to tune the model for particular applications. 



Configuring the model and tokenizer 

We start by specifying the pre-trained Llama 2 model and prepare for an improved version called “llama-2-7b-enhanced“. We load the tokenizer and make slight adjustments to ensure compatibility with half-precision floating-point numbers (fp16) operations. Working with fp16 can offer various advantages, including reduced memory usage and faster model training. However, it’s important to note that not all operations work seamlessly with this lower precision format, and tokenization, a crucial step in preparing text data for model training, is one of them. 


Next, we load the pre-trained Llama 2 model with our quantization configurations. We then deactivate caching and configure a pretraining temperature parameter. 


In order to shrink the model’s size and boost inference speed, we employ 4-bit quantization provided by the BitsAndBytesConfig. Quantization involves representing the model’s weights in a way that consumes less memory. 


The configuration mentioned here uses the ‘nf4‘ type for quantization. You can experiment with different quantization types to explore potential performance variations. 



Quantization configuration 

In the context of training a machine learning model using Low-Rank Adaptation (LoRA), several parameters play a significant role. Here’s a simplified explanation of each: 



Parameters specific to LoRA: 


  • Dropout Rate (lora_dropout): This parameter represents the probability that the output of each neuron is set to zero during training. It is used to prevent overfitting, which occurs when the model becomes too tailored to the training data. 


  • Rank (r): Rank measures how the original weight matrices are decomposed into simpler, smaller matrices. This decomposition reduces computational demands and memory usage. Lower ranks can make the model faster but may impact its performance. The original LoRA paper suggests starting with a rank of 8, but for QLoRA, a rank of 64 is recommended. 


  • Lora_alpha: This parameter controls the scaling of the low-rank approximation. It’s like finding the right balance between the original model and the low-rank approximation. Higher values can make the approximation more influential during the fine-tuning process, which can affect both performance and computational cost. 


By adjusting these parameters, particularly lora_alpha and r, you can observe how the model’s performance and resource utilization change. This allows you to fine-tune the model for your specific task and find the optimal configuration. 



You can find the code of this notebook here. 


I asked both the fine-tuned and unfine-tuned models of LLaMA 2 about a university, and the fine-tuned model provided the correct result. The unfine-tuned model does not know about the query therefore it hallucinated the response. 

Unfine tuned


fine tuned