Python libraries

Python is a programming language that has become the backbone of modern AI and machine learning. It provides the perfect mix of simplicity and power, making it the go-to choice for AI research, deep learning, and Generative AI.

Python plays a crucial role in enabling machines to generate human-like text, create realistic images, compose music, and even design code. From academic researchers and data scientists to creative professionals, anyone looking to harness AI’s potential uses Python to boost their skills and build real-world applications.

But what makes Python so effective for Generative AI?

The answer lies in Python libraries which are specialized toolkits that handle complex AI processes like deep learning, natural language processing, and image generation. Understanding these libraries is key to unlocking the full potential of AI-driven creativity.

In this blog, we’ll explore the top Python libraries for Generative AI, breaking down their features, use cases, and how they can help you build the next big AI-powered creation. Let’s begin with understanding what Python libraries are and why they matter.

 

 

What are Python Libraries?

When writing code for a project, it is a great help if you do not have to write every single line of code from scratch. This is made possible by the use of Python libraries.

A Python library is a collection of pre-written code modules that provide specific functionalities, making it easier for developers to implement various features without writing the code all over again. These libraries bundle useful functions, classes, and pre-built algorithms to simplify complex programming tasks.

Whether you are working on machine learning, web development, or automation, Python libraries help you speed up development, reduce errors, and improve efficiency. These libraries are one of the most versatile and widely used programming tools.

 

Here’s a list of useful Python packages that you must know about

 

Here’s why they are indispensable for developers:

Code Reusability – Instead of writing repetitive code, you can leverage pre-built functions, saving time and effort.

Simplifies Development – Libraries abstract away low-level operations, so you can focus on higher-level logic rather than reinventing solutions.

Community-Driven & Open-Source – Most Python libraries are backed by large developer communities, ensuring regular updates, bug fixes, and extensive documentation.

Optimized for Performance – Libraries like NumPy and TensorFlow are built with optimized algorithms to handle complex computations efficiently.

Popular Python Libraries for Generative AI

Python is a popular programming language for generative AI, as it has a wide range of libraries and frameworks available. Here are 10 of the top Python libraries for generative AI:

 1. TensorFlow

Developed by Google Brain, TensorFlow is an open-source machine learning (ML) library that makes it easy to build, train, and deploy deep learning models at scale. It simplifies the entire ML pipeline, from data preprocessing to model optimization.

TensorFlow provides robust tools and frameworks for building and training generative models, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). It can be used to train and deploy a variety of generative models, such as GANs, autoencoders, diffusion models, and more.

Here’s a list of the types of neural networks

The TensorFlow library provides:

• TensorFlow Hub – A collection of ready-to-use models for quick experimentation.
• Colab Notebooks – A beginner-friendly way to run TensorFlow code in the cloud without installations.
• TensorFlow.js – Bring AI into web applications with JavaScript support.
• TensorFlow Lite – Deploy AI models on mobile devices and edge computing for real-world applications.
• TensorFlow Extended (TFX) – A complete suite for building production-grade AI models, ensuring seamless deployment.
• Keras Integration – Offers an intuitive API that simplifies complex AI model building, making it accessible to beginners and pros alike.

This makes TensorFlow a good choice for generative AI because it is flexible and powerful with a large community of users and contributors. Thus, it remains at the forefront, enabling developers, artists, and innovators to push the boundaries of what AI can create. If you are looking to build the next AI-powered masterpiece, TensorFlow is your ultimate tool.

 

 

2. PyTorch

PyTorch is another popular open-source machine learning library that is well-suited for generative AI. It has been developed by Meta AI (Facebook AI Research), becoming a popular tool among researchers, developers, and AI enthusiasts.

What makes PyTorch special?

It combines flexibility, ease of use, and unmatched performance, making it the go-to library for Generative AI applications. Whether you’re training neural networks to create images, synthesize voices, or generate human-like text, PyTorch gives you the tools to innovate without limits.

It is a good choice for beginners and experienced users alike, enabling all to train and deploy a variety of generative models, like conditional GANs, autoregressive models, and diffusion models. Below is a list of features PyTorch offers to make it easier to deploy AI models:

• TorchVision & TorchAudio – Ready-to-use datasets and tools for AI-powered image and audio processing.
• TorchScript for Production – Convert research-grade AI models into optimized versions for real-world deployment.
• Hugging Face Integration – Access pre-trained transformer models for NLP and AI creativity.
• Lightning Fast Prototyping – Rapidly build and test AI models with PyTorch Lightning.
• CUDA Acceleration – Seamless GPU support ensures fast and efficient model training.
• Cloud & Mobile Deployment – Deploy your AI models on cloud platforms, mobile devices, or edge computing systems.

PyTorch is a good choice for generative AI because it is easy to use and has a large community of users and contributors. It empowers developers, artists, and innovators to create futuristic AI applications that redefine creativity and automation.

 

 

3. Transformers

Transformers is a Python library by Hugging Face that provides a unified API for training and deploying transformer models. Transformers are a type of neural network architecture that is particularly well-suited for natural language processing tasks, such as text generation and translation.

If you’ve heard of GPT, BERT, T5, or Stable Diffusion, you’ve already encountered the power of transformers. They can be used to train and deploy a variety of generative models, including transformer-based text generation models like GPT-3 and LaMDA.

Instead of training models from scratch (which can take weeks), Transformers lets you use and fine-tune powerful models in minutes. Its key features include:

• Pre-Trained Models – Access 1000+ AI models trained on massive datasets.
• Multi-Modal Capabilities – Works with text, images, audio, and even code generation.
• Easy API Integration – Get AI-powered results with just a few lines of Python.
• Works Across Frameworks – Supports TensorFlow, PyTorch, and JAX.
• Community-Driven Innovation – A thriving community continuously improving the library.

Transformers is a good choice for generative AI because it is easy to use and provides a unified API for training and deploying transformer models. It has democratized Generative AI, making it accessible to anyone with a vision to create.

4. Diffusers

Diffusers is a Python library for diffusion models, which are a type of generative model that can be used to generate images, audio, and other types of data. Developed by Hugging Face, this library provides a seamless way to create stunning visuals using generative AI.

Diffusers provides a variety of pre-trained diffusion models and tools for training and fine-tuning your own models. Such models will excel at generating realistic, high-resolution images, videos, and even music from noise.

 

Explore the RAG vs Fine-tuning debate

 

Its key features can be listed as follows:

• Pre-Trained Diffusion Models – Includes Stable Diffusion, Imagen, and DALL·E-style models.
• Text-to-Image Capabilities – Convert simple text prompts into stunning AI-generated visuals.
• Fine-Tuning & Custom Models – Train or adapt models to fit your unique creative vision.
• Supports Image & Video Generation – Expand beyond static images to AI-powered video synthesis.
• Easy API & Cross-Framework Support – Works with PyTorch, TensorFlow, and JAX.

Diffusers is a good choice for generative AI because it is easy to use and provides a variety of pre-trained diffusion models. It is at the core of some of the most exciting AI-powered creative applications today because Diffusers gives you the power to turn ideas into visual masterpieces.

 

 

 

5. Jax

Jax is a high-performance numerical computation library for Python with a focus on machine learning and deep learning research. It is developed by Google AI and has been used to achieve state-of-the-art results in a variety of machine learning tasks, including generative AI.

It is an alternative to NumPy with automatic differentiation, GPU/TPU acceleration, and parallel computing capabilities. Jax brings the power of automatic differentiation and just-in-time (JIT) compilation to Python.

It’s designed to accelerate machine learning, AI research, and scientific computing by leveraging modern hardware like GPUs and TPUs seamlessly. Some key uses of Jax for generative AI include training GANs, diffusion models, and more.

At its core, JAX provides:

• NumPy-like API – A familiar interface for Python developers.
• Automatic Differentiation (Autograd) – Enables gradient-based optimization for deep learning.
• JIT Compilation (via XLA) – Speeds up computations by compiling code to run efficiently on GPUs/TPUs.
• Vectorization (via vmap) – Allows batch processing for large-scale AI training.
• Parallel Execution (via pmap) – Distributes computations across multiple GPUs effortlessly.

In simple terms, JAX makes your AI models faster, more scalable, and highly efficient, unlocking performance levels beyond traditional deep learning frameworks.

 

Get started with Python, check out our instructor-led Python for Data Science training.

 

6. LangChain

LangChain is a Python library for chaining multiple generative models together. This can be useful for creating more complex and sophisticated generative applications, such as text-to-image generation or image-to-text generation. It helps developers chain together multiple components—like memory, APIs, and databases—to create more dynamic and interactive AI applications.

This library is a tool for developing applications powered by large language models (LLMs). It acts as a bridge, connecting LLMs like OpenAI’s GPT, Meta’s LLaMA, or Anthropic’s Claude with external data sources, APIs, and complex workflows.

If you’re building chatbots, AI-powered search engines, document processing systems, or any kind of generative AI application, LangChain is your go-to toolkit. Key features of LangChain include:

• Seamless Integration with LLMs – Works with OpenAI, Hugging Face, Cohere, Anthropic, and more.
• Memory for Context Retention – Enables chatbots to remember past conversations.
• Retrieval-Augmented Generation (RAG) – Enhances AI responses by fetching real-time external data.
• Multi-Agent Collaboration – Enables multiple AI agents to work together on tasks.
• Extensive API & Database Support – Connects with Google Search, SQL, NoSQL, vector databases, and more.
• Workflow Orchestration – Helps chain AI-driven processes together for complex automation.

Hence, LangChain supercharges LLMs, making them more context-aware, dynamic, and useful in real-world applications.

 

Learn all you need to know about what is LangChain

 

7. LlamaIndex

In the world of Generative AI, one of the biggest challenges is connecting AI models with real-world data sources. LlamaIndex is the bridge that makes this connection seamless, empowering AI to retrieve, process, and generate responses from structured and unstructured data efficiently.

LlamaIndex is a Python library for ingesting and managing private data for machine learning models. It can be used to store and manage your training datasets and trained models in a secure and efficient way. Its key features are:

• Data Indexing & Retrieval – Organizes unstructured data and enables quick, efficient searches.
• Seamless LLM Integration – Works with GPT-4, LLaMA, Claude, and other LLMs.
• Query Engine – Converts user questions into structured queries for accurate results.
• Advanced Embeddings & Vector Search – Uses vector databases to improve search results.
• Multi-Source Data Support – Index data from PDFs, SQL databases, APIs, Notion, Google Drive, and more.
• Hybrid Search & RAG (Retrieval-Augmented Generation) – Enhances AI-generated responses with real-time, contextual data retrieval.

This makes LlamaIndex a game-changer for AI-driven search, retrieval, and automation. If you want to build smarter, context-aware AI applications that truly understand and leverage data, it is your go-to solution.

 

Read in detail about the LangChain vs LlamaIndex debate

 

8. Weight and Biases

Weights & Biases is an industry-leading tool for experiment tracking, hyperparameter optimization, model visualization, and collaboration. It integrates seamlessly with popular AI frameworks, making it a must-have for AI researchers, ML engineers, and data scientists.

Think of W&B as the control center for your AI projects, helping you track every experiment, compare results, and refine models efficiently. Below are some key features of W&B:

• Experiment Tracking – Log model parameters, metrics, and outputs automatically.
• Real-Time Visualizations – Monitor losses, accuracy, gradients, and more with interactive dashboards.
• Hyperparameter Tuning – Automate optimization with Sweeps, finding the best configurations effortlessly.
• Dataset Versioning – Keep track of dataset changes for reproducible AI workflows.
• Model Checkpointing & Versioning – Save and compare different versions of your model easily.
• Collaborative AI Development – Share experiment results with your team via cloud-based dashboards.

Hence, if you want to scale your AI projects efficiently, Weights & Biases is a must-have tool. It eliminates the hassle of manual logging, visualization, and experiment tracking, so you can focus on building groundbreaking AI-powered creations.

 

 

The Future of Generative AI with Python

Generative AI is more than just a buzzword. It is transforming the way we create, innovate, and solve problems. Whether it is AI-generated art, music composition, or advanced chatbots, Python and its powerful libraries make it all possible.

What’s exciting is that this field is evolving faster than ever. New tools, models, and breakthroughs are constantly pushing the limits of what AI can do.

 

 

And the best part?

Most of these advancements are open-source, meaning anyone can experiment, build, and contribute. So, if you’ve ever wanted to dive into AI and create something groundbreaking, now is the perfect time. With Python by your side, the possibilities are endless. The only question is: what will you build next?

What is web scraping?

Web scraping is the act of extracting the content and data from a website. The vast amount of data available on the internet is not open and available to download. As a result, ethical web scraping is the most effective technique to collect this data. There is also a debate about the legality of web scraping as the content may get stolen or the website can crash as a result of web scraping.

Ethical Web Scraping is the act of harvesting data legally by following ethical rules about web scraping. There are certain rules in ethical web scraping that when followed ensure trust between the website owner and web scraper.

Web scraping using Python

In Python, a learner can write a small piece of code to do large tasks. Since web scraping is used to save time, a small code written in Python can save a lot of time. Also, Python is simple and easy to understand and provides an extensive set of libraries for web scraping and further manipulation required on extracted data.

PRO TIP: Join our 5-day instructor-led Python for Data Science training to enhance your web scraping skills.

Challenges for individuals

Individuals who are new to web scraping and wish to flourish in their field usually lack the necessary computing and learning resources to obtain hands-on expertise. Also, they may face compatibility issues when installing libraries.

What we provide

With just a single click, Jupyter Hub for Ethical Web Scraping using Python comes with pre-installed Web Scraping python libraries, which gives the learner an effortless coding environment in the Azure cloud and reduces the burden of installation. Moreover, this offer provides the learner with a repository of the famous book on web scraping which contains chapter-wise notebooks which serve as a learning resource for a user in gaining hands-on experience with web scraping.

Through this offer, a learner can collect data from various sources legally by following the best practices for ethical web scraping mentioned in the latter section of this blog. Once the data is collected, it can be further analyzed to get valuable insights into almost everything while all the heavy computations are performed on Microsoft Azure hence saving the user from the trouble of running high computations on the local machine.

Python libraries:

Listed below are the pre-installed web scraping python libraries and the sources of repositories of web scraping book provided by this offer:

•          Pandas
•          NumPy
•          Scikit-learn
•          Beautifulsoup4
•          lxml
•          MechanicalSoup
•          Requests
•          Scrapy
•          Selenium
•          urllib

Repository:

•          GitHub repository of book ‘Web Scraping with Python 2nd Edition’,
  by author Ryan Mitchell.

Best practices for ethical web scraping

Globally, there is a debate about whether web scraping is an ethical concept or not. The reason it is unethical is that when a website is queried repeatedly by the same user (in this case bot), too many requests land on the server simultaneously and all resources of the server may be consumed in generating responses for each request, preventing it from responding to other legitimate users.

In this way, the server denies responses to any further users, commonly known as a Denial of Service (DoS) attack.

Below are the best practices for ethical web scraping, and compliance with these will allow a web scraper to work ethically.

1.   Check out for ROBOTS.TXT

Robots.txt file, also known as the Robots Exclusion Standard, is used to inform the web scrapers if the website can be crawled or not, if yes then how to index the website. A legitimate web scraper is expected to respect the instructions in this file and not disobey the website owner’s allowed instructions.

2.   Check for website APIs

An ethical web scraper is expected to first look for the public API of the website in question instead of scraping it all together. Many website owners provide public API access which can be used by anyone looking to gain from the information available on the website. Provision of public API works in the best interests of both the ethical scrapper as well as the website owner, avoiding web scraping altogether.

3.   Avoid repeated requests

Vigorous scraping can occasionally cause functionality issues, resulting in a poor user experience for humans. As a result, it is always advised to scrape during off-peak hours. An ethical web scraper is expected to delay recurrent requests to avoid a DoS attack.

4.   Provide your identity

It is always a good idea to take responsibility for one’s actions. An ethical web scraper never hides his or her identity and provides it in a user-agent string. Not only does this make the intentions of the scraper clear but also provides a means of contact for any questions or concerns of the website owner.

5.   Avoid fake ownership

The content scraped through web scraper should always be respected and never passed on under the fake information of scraper as the author. This act can be regarded as highly unethical as well as illegal since the website owner may file a copyright claim. It also damages the reputation of genuine web scrapers and hurts the trust of the website owner.

6.  Ask for permission

Since the website information belongs to the owner, one should never presume it to be free and ask politely to use it for their means. An ethical web scraper always seeks permission from the website owner to avoid any future problems. The website owner should be given the choice of whether she agrees to scrape the data.

 7.  Give due credit

To encourage the website owner as a token of thanks, the web scraper should give due credit wherever possible. This can be done in many ways such as providing a link to the original website on any blog, article, or social media post by generating traffic for the original website.

Conclusion

Ethical web scraping is a two-way street in which the website owner should be mindful of the global availability of the data, similarly, the scraper should not harm the website in any way and also first seek permission from the website owner. If a web scraper abides by the above-mentioned practices, I.e., he/she works ethically, the web owner may not only allow scraping his or her website but also provide helpful means to the scraper in the form of Meta data or a public API.

At Data Science Dojo, we deliver data science education, consulting, and technical services to increase the power of data. We are therefore adding a free Jupyter Notebook Environment dedicated specifically for Ethical Web Scraping using Python. Install the Jupyter Hub offer now from the Azure Marketplace by Data Science Dojo, your ideal companion in your journey to learn data science!

What is data visualization?

Data Science Dojo has launched Jupyter Hub for Machine Learning using Python offering to the Azure Marketplace with pre-installed machine learning libraries and pre-cloned GitHub repositories of famous machine learning books which help the learner to take the first steps into the field of machine learning.

What is machine learning?

Machine learning is a sub-field of Artificial Intelligence. It is an innovative technology that allows machines to learn from historical data and provide the best results to predict outcomes.

Machine learning using Python

Machine learning requires exploratory data analysis, data processing, and the training of data to predict outcomes. Python provides a vast number of libraries and frameworks that let the user collect, analyze and transform data by just using built-in functions provided by the library which makes coding easy and also saves a significant amount of time.

 PRO TIP: Join our 5-day instructor-led Python for Data Science training to enhance your machine learning skills.

Challenges for individuals

Individuals who are new to machine learning and want to excel in their path in machine learning usually lack computing as well as learning resources to gain hands-on experience with machine learning. A beginner in machine learning also faces compatibility issues while installing libraries.

What we provide

With just a single click, Jupyter Hub for Machine Learning using Python comes with pre-installed machine learning python libraries, which gives the learner an effortless coding environment in the Azure cloud and reduces the burden of installation. Moreover, this offer provides the learner with repositories of famous books on machine learning which contain chapter-wise notebooks which serve as a learning resource for a user in gaining hands-on experience with machine learning. The heavy computations required for Machine Learning applications are not performed on the user’s local machine. Instead, they are performed in the Azure cloud, which increases responsiveness and processing speed.

Listed below are the pre-installed machine learning python libraries and the sources of repositories of machine learning books provided by this offer:

Python libraries

• Pandas
• NumPy
• scikit-learn
• mlpack
• matplotlib
• SciPy
• Theano
• Pycaret
• Orange3
• seaborn

Repositories

•  Github repository of book ‘Python Machine Learning Book 1st Edition’, by author Sebastian Raschka.
•  Github repository of book ‘Python Machine Learning Book 2nd Edition’, by author Sebastian Raschka.
•  Github repository of the book ‘Hands-on Machine Learning with Scikit Learn, Keras, and TensorFlow’, by author Geron-Aurelien.
•  Github repository of ‘Microsoft Azure Cloud Advocates 12-week Machine Learning curriculum’.

Conclusion

Jupyter Hub for Machine Learning using Python provides an in-browser coding environment with just a single click, hence providing ease of installation. Through this offer, a user can work on a variety of machine learning applications including stock market trading, email spam and malware filtering, product recommendations, online customer support, medical diagnosis, online fraud detection, and image recognition.

Jupyter Hub for Machine Learning using Python offered by Data Science Dojo is ideal to learn more about machine learning without the need to worry about configurations and computing resources. The heavy resource requirement for processing and training large data for these applications is no longer an issue as data-intensive computations are now performed on Microsoft Azure which increases processing speed.

At Data Science Dojo, we deliver data science education, consulting, and technical services to increase the power of data. We are therefore adding a free Jupyter Notebook Environment dedicated specifically for Machine Learning using Python. The offering leverages the power of Microsoft Azure services to run effortlessly with outstanding responsiveness. Install the Jupyter Hub offer now from the Azure Marketplace by Data Science Dojo, your ideal companion in your journey to learn data science!

Data Science Dojo has launched  Jupyter Hub for Computer Vision using Python offering to the Azure Marketplace with pre-installed libraries and pre-cloned GitHub repositories of famous Computer Vision books and courses which enables the learner to run the example codes provided.

Out of all the Python scientific libraries and packages available, which ones are not only popular but the most useful in getting the job done?

Python packages for data science and libraries

To help you filter down a list of libraries and packages worth adding to your data science toolbox, we have compiled our top picks for aspiring and practicing data scientists. But you’ll also want to know how to best use these tools for tricky, real-world data problems. So instead of leaving you with yet another top-choice list among a quintillion list, we explain how to make the most of these libraries using real-world examples.

You can learn more about how these packages fit into data science with Data Science Dojo’s introduction to Python course.

Data manipulation

Pandas

There’s a reason why pandas consistently tops published ranks on data science-related libraries in Python. The library can help you with a variety of tasks, but it is particularly useful for data manipulation or data wrangling. It can save you a lot of leg work in not only your typical rudimentary data manipulation tasks but also in handling some pretty tricky problems you might encounter when slicing and filtering.

Multi-indexed data can be one of these tricky tasks. The library pandas takes care of advanced indexing, including multi-indexing, where you might need to work with higher-dimensional data or multiple index levels. For example, the number of user interactions might be indexed by 1) product category, 2) the time of day the user interacted with the product, and 3) the location of the user.

Instead of your typical table of rows and columns to represent the data, you might find it better to organize the number of user interactions into all cases that fall under the x product category, with y time of day, and z location. This way you can easily see user interactions across each condition of product category, time of day, and user location. This saves you from having to apply a filter or group for all combinations of conditions in your traditional row-and-table structure.

Here is one way to multi-index data in pandas. With less than a few lines of code, pandas makes this easy to implement in Python:

import pandas as pd

data_multi_indx = table_data.set_index(['Product', 'Day of Week'])
print(data_multi_indx)
'''
Output:
                      Location  Num User Interactions
Product   Day of Week
Product 1 Morning            A                      3
          Morning            B                     90
          Morning            C                      7
          Afternoon          A                     17
          Afternoon          B                      1
          Afternoon          C                     82
Product 2 Morning            A                     27
          Morning            B                     70
          Morning            C                      3
          Afternoon          A                      1
          Afternoon          B                      1
          Afternoon          C                     98
Product 3 Morning            A                     94
          Morning            B                      5
          Morning            C                      1
          Afternoon          A                      0
          Afternoon          B                      7
          Afternoon          C                     93
'''

For the more rudimentary data manipulation tasks, pandas doesn’t require much effort on your part. You can simply use the functions available for imputing missing values, one-hot encoding, dropping columns and rows, and so on.

Here are a few example classes and functions  pandas that make rudimentary data manipulation easy in a few lines of code, at most.

For more lessons with Pandas, visit Data Independent.

Feature	Description
fillna(value)	Fill in missing values on a column or the whole data frame with a value such as the mean, median, or mode.
isna(data)/isnull(data)	Check for missing values.
get_dummies(data_frame['Column'])	Apply one-hot encoding on a column.
to_numeric(data_frame['Column'])	Convert a column of values from strings to numeric values.
to_string(data_frame['Column'])	Convert a column of values from numeric values to strings.
to_datetime(data_frame['Column'])	Convert a column of datetimes in string format to standard datetime format.
drop(columns=['Column0','Column1'])	Drop specific columns or useless columns in your data frame.
drop(data.frame.index[[rownum0,rownum1]])	Drop specific rows or useless rows in your data frame.

NumPy

Another library that keeps topping the ranks is numpy. This library can handle many tasks, but it is particularly useful when working with multi-dimensional arrays and performing calculations on these arrays. This can be tricky to do in more conventional ways, where you need to find the index of a value or certain values inside another index, with multiple indices.

 

Read about Top Python projects to choose from in 2023

 

This is where numpy shows its strength. Its array() function means standard arrays can be simply added and nicely bundled into a multi-dimensional array. Calculations on these arrays can also be easily implemented using numpy’s vast array (pun intended) of mathematical functions.

Let’s picture an example where numpy’s multi-dimensional arrays are useful. A company tracks or records if a user was/was not shown a mobile product in the morning, afternoon, and night, delivered through a mobile notification. Based on the level of user interaction with the shown product, the company also records a user engagement score.

Data points on each user’s shown product and engagement score are stored inside an array; each array stores these values for each user. The company would like to quickly and simply bundle all user arrays.

In addition to this, using engagement score and purchase history, the company would like to calculate and identify the minimum distance (or difference) across all users’ data points so that users who follow a similar pattern can be categorized and targeted accordingly.

numpy’s array() makes it easy to bundle user arrays into a multi-dimensional array and argmin() and linalg.norm() find the min Euclidean distance between users, as an example of the kinds of calculations that can be done on a multi-dimensional array:

import numpy as np

# Records tracking whether user was/was not shown product during
# morning, afternoon, and night, and user engagement score
user_0 = [0,0,1,0.7]
user_1 = [0,1,0,0.4]
user_2 = [1,0,0,0.0]
user_3 = [0,0,1,0.9]
user_4 = [0,1,0,0.3]
user_5 = [1,0,0,0.0]
# Create a multi-dimensional array to bundle all users
# Can use arrays with mixed data types by specifying 
# the object data type in numpy multi-dimensional arrays
users_multi_dim = np.array([user_0,user_1,user_2,user_3,user_4,user_5],dtype=object)
print(users_multi_dim)
'''
Output:
[[0 0 1 0.7]
 [0 1 0 0.4]
 [1 0 0 0.0]
 [0 0 1 0.9]
 [0 1 0 0.3]
 [1 0 0 0.0]]
'''
# To view which user was/was not shown the product
# either morning, afternoon or night, pandas easily 
# allows you to index and label the data
row_names = [_ for _ in ['User 0','User 1','User 2','User 3','User 4','User 5']]
col_names = [_ for _ in ['Product Shown Morning','Product Shown Afternoon',
                         'Product Shown Night','User Engagement Score']]
users_df_indexed = pd.DataFrame(users_multi_dim,index=row_names,columns=col_names)
print(users_df_indexed)
'''
Output:
       Product Shown Morning Product Shown Afternoon Product Shown Night User Engagement Score
User 0                     0                       0                   1                   0.7
User 1                     0                       1                   0                   0.4
User 2                     1                       0                   0                     0
User 3                     0                       0                   1                   0.9
User 4                     0                       1                   0                   0.3
User 5                     1                       0                   0                     0
'''
# Find which existing user is closest to the engagement 
# and purchase behavior of a new user by calculating the 
# min Euclidean distance on a numpy multi-dimensional array
user_0 = [0.7,51.90,2]
user_1 = [0.4,25.95,1]
user_2 = [0.0,0.00,0]
user_3 = [0.9,77.85,3]
user_4 = [0.3,25.95,1]
user_5 = [0.0,0.00,0]
users_multi_dim = np.array([user_0,user_1,user_2,user_3,user_4,user_5])
new_user = np.array([0.8,77.85,3])
closest_to_new = np.argmin(np.linalg.norm(users_multi_dim-new_user,axis=1))
print('User', closest_to_new, 'is closest to the new user')
'''
Output:
User 3 is closest to the new user
'''

Data modeling

Statsmodels

The main strength of statsmodels is its focus on statistics, going beyond the ‘machine learning out-of-the-box’ approach. This makes it a popular choice for data scientists. Conducting statistical tests to find significantly different variables, checking for normality in your data, checking the standard errors, and so on, cannot be underestimated when trying to build the most effective model you can build. Your model is only as good as your inputs, and statsmodels is designed to help you better understand and customize your inputs.

The library also covers an exhaustive list of predictive models to choose from, depending on your predictors and outcome variable(s). It covers your classic Linear Regression models (including ordinary least squares, weighted least squares, recursive least squares, and more), Generalized Linear models, Linear Mixed Effects models, Binomial and Poisson Bayesian models, Logit and Probit models, Time Series models (including autoregressive integrated moving average, dynamic factor, unobserved component, and more), Hidden Markov models, Principal Components and other techniques for Multivariate models, Kernel Density estimators, and lots more.

Here are the classes and functions in statsmodels that cover the main modeling techniques useful for many prediction tasks.

Scikit-learn

Any library that makes machine learning more accessible and easier to implement is bound to make the top choice list among aspiring and practicing data scientists. The Library scikit-learn not only allows models to be easily implemented out-of-the-box but also offers some auto fine-tuning.

Finding the best possible combination of model parameters is a key example of fine-tuning. The library offers a few good ways to search for the optimal set of parameters, given the algorithm and problem to solve. The grid search and random search algorithms in scikit-learn evaluate different combinations of parameters until they find the best combo that results in the best outcome, or a better-performing model.

The grid search goes through every possible combination, whereas the random search randomly samples the parameters over a fixed number of times/iterations. Cross-validating your model on many subsets of data is also easy to implement using scikit-learn. With this kind of automation, the library offers data scientists a massive time saver when building models.

The library also covers all the essential machine learning models from classification (including Support Vector Machine, Random Forest, etc), to regression (including Ridge Regression, Lasso Regression, etc), and clustering (including k-Means, Mean Shift, etc).

Here are the classes and functions in scikit-learn that cover the main modeling techniques useful for many prediction tasks.

Feature	Description
SVC(), GaussianNB(), LogisticRegression(), DecisionTreeClassifier(), RandomForestClassifier(), SGDClassifier(), MLPClassifier()	Classification models: Support Vector Machine, Gaussian Naïve Bayes, Logistic Regression, Decision Tree, Random Forest, Stochastic Gradient Descent, Multi-Layer Perceptron
linear_model.Ridge(), linear_model.Lasso(), SVR(), DecisionTreeRegressor(), RandomForestRegressor(), SGDRegressor, MLPRegressor()	Regression models: Ridge Regression, Lasso Regression, Support Vector Machine, Decision Tree, Random Forest, Stochastic Gradient Descent, Multi-Layer Perceptron
KMeans(), AffinityPropagation(), MeanShift(), AgglomerativeClustering	Clustering models: k-Means, Affinity Propagation, Mean Shift, Agglomerative Hierarchical Clustering

Data visualization

Plotly

The libraries matplotlib and seaborn will easily take care of your basic static plot functions, which are important for your own internal exploration or understanding of the data. But when presenting visual insights to business folks or users, interactivity is where we are headed these days.

Using JavaScript functionality, plotly renders interactive graphs in the form of zooming in and panning out of the graph panel, hovering over objects for more information, and dragging objects into position to further explore relationships in the data. Graphs can be customized to your heart’s content.

Here are just a few of the many tricks that Plotly offers:

Feature	Description
hovermode, hoverinfo	Controls the mode and text when a user hovers over an object.
on_selection(), on_click()	Allows a user to select or click on an object and have that selected object change color, for example.
update	Modifies a graph’s layout and data such as titles and annotations.
animate	Creates an animated graph.

Bokeh

Much like plotly, bokeh also offers interactive graphs. But one feature that stands out in bokeh is linked interactions. This is useful when keeping separate graphs in unison, where the user interacts with one graph and needs to compare with the other while they are in sync. For example, a user zooms into a graph, effectively changing the range of the graph, and then would like to compare it with the second graph. The second graph would need to automatically update its range so that both graphs can be easily compared like-for-like.

Here are some key tricks that bokeh offers:

Feature	Description
figure()	Creates a new plot and allows linking to the range of another plot.
HoverTool(), hover_glyph	Allows users to hover over an object for more information.
selection_glyph	Selects a particular glyph object for styling.
Slider()	Creates a slider to dynamically update the plot based on the slide range.