Large Language Models Bootcamp

Use LLMs to make decisions about what to do next. Enable these decisions with tools. In this module, we’ll talk about agents. We’ll learn what they are, how they work, and how to use them within the LangChain library to superpower our LLMs.

• Agents and Tools
• Agent Types
  • Conversational agents
  • OpenAI functions agents
  • ReAct agents
  • Plan and execute agents
• Hands-on Exercise: Create and execute some of the following agents
  • Excel agent
  • JSON agent
  • Python Pandas agent
  • Document comparison agent
  • Power BI agent

LLMOps encompasses the practices, techniques and tools used for the operational management of large language models in production environments. LLMs offer tremendous business value, humans are involved in all stages of the lifecycle of an LLM from acquisition of data to interpretation of insights. In this module we will learn about the following:

• Principles of Responsible AI
  • Fairness and Eliminating Bias
  • Reliability and Safety
  • Privacy and Data Protection
• Review techniques for assessing large language model applications, including:
  • Model fine-tuning
  • Model inference and serving
  • Model monitoring with human feedback
• Introduce LangKit by WhyLabs for data-centric LLMOps:
  • Guardrails: Define rules to govern prompts and responses for LLM applications.
  • Evaluation: Assess LLM performance using known prompts to identify issues.
  • Observability: Collect telemetry data about the LLM's internal state for monitoring and issue detection.
• Hands-on Exercise:
  • Using Langkit Evaluate LLM performance on specific prompts

Dive into Large Language Model (LLM) evaluation, examining its importance, common issues, and key metrics such as BLEU and ROUGE, and apply these insights through a hands-on summarization exercise.

Introduction to LLM Evaluation

• What is evaluation and why is it important for LLMs?
• Overview of common mistakes made by LLMs
• Brief introduction to benchmark datasets and metrics

Evaluation Metrics

• Explain commonly used automatic metrics (BLEU, ROUGE, BERTScore)
• Compare strengths and weaknesses of different metrics
• Discuss role of human evaluation and techniques (Likert scale)

Hands-on Exercise

• Evaluating LLMs summarization using metrics like Rouge and Bertscore

This module covers how to scale and automate LLM applications using ZenML. ZenML streamlines data versioning, caching, deployment, and collaboration for efficient LLM app development.

Key Challenges in building Enterprise-Level LLM Apps

• Introduction about challenges in production
• Data Versioning in production
• Overview of ZenML's role in scaling LLM apps
• Dashboard access for tracking pipeline progress and data storage

Hands-On Exercise

• Create a QnA agent with ZenML pipelines