analytics

In today’s dynamic digital world, handling vast amounts of data across the organization is challenging. It takes a lot of time and effort to set up different resources for each task and duplicate data repeatedly. Picture a world where you don’t have to juggle multiple copies of data or struggle with integration issues.

Microsoft Fabric makes this possible by introducing a unified approach to data management. Microsoft Fabric aims to reduce unnecessary data replication, centralize storage, and create a unified environment with its unique data fabric method. 

What is Microsoft Fabric?

Microsoft Fabric is a cutting-edge analytics platform that helps data experts and companies work together on data projects. It is based on a SaaS model that provides a unified platform for all tasks like ingesting, storing, processing, analyzing, and monitoring data.

With this full-fledged solution, you don’t have to spend all your time and effort combining different services or duplicating data.

Fabric features a lake-centric architecture, with a central repository known as OneLake. OneLake, being built on Azure Data Lake Storage (ADLS), supports various data formats, including Delta, Parquet, CSV, and JSON. OneLake offers a unified data environment for each of Microsoft Fabric’s experiences.

These experiences facilitate professionals from ingesting data from different sources into a unified environment and pipelining the ingestion, transformation, and processing of data to developing predictive models and analyzing the data by visualization in interactive BI reports.  

Microsoft Fabric’s experiences include: 

• Synapse Data Engineering 
• Synapse Data Warehouse 
• Synapse Data Science 
• Synapse Real-Time Intelligence 
• Data Factory 
• Data Activator  
• Power BI

Exploring Microsoft Fabric Components: Sales Use Case

Microsoft Fabric offers a set of analytics components that are designed to perform specific tasks and work together seamlessly. Let’s explore each of these components and its application in the sales domain: 

Synapse Data Engineering:

Synapse Data Engineering provides a powerful Spark platform designed for large-scale data transformations through Lakehouse.

In the sales use case, it facilitates the creation of automated data pipelines that handle data ingestion and transformation, ensuring that sales data is consistently updated and ready for analysis without manual intervention.

Synapse Data Warehouse:

Synapse Data Warehouse represents the next generation of data warehousing, supporting an open data format. The data is stored in Parquet format and published as Delta Lake Logs, supporting ACID transactions and enabling interoperability across Microsoft Fabric workloads.

In the sales context, this ensures that sales data remains consistent, accurate, and easily accessible for analysis and reporting. 

Synapse Data Science:

Synapse Data Science empowers data scientists to work directly with secured and governed sales data prepared by engineering teams, allowing for the efficient development of predictive models.

By forecasting sales performance, businesses can identify anomalies or trends, which are crucial for directing future sales strategies and making informed decisions.

Synapse Real-Time Intelligence:

Real-Time Intelligence in Synapse provides a robust solution to gain insights and visualize event-driven scenarios and streaming data logs. In the sales domain, this enables real-time monitoring of live sales activities, offering immediate insights into performance and rapid response to emerging trends or issues.  

Data Factory:

Data Factory enhances the data integration experience by offering support for over 200 native connectors to both on-premises and cloud data sources.

For the sales use case, this means professionals can create pipelines that automate the process of data ingestion, and transformation, ensuring that sales data is always updated and ready for analysis.  

Data Activator:

Data Activator is a no-code experience in Microsoft Fabric that enables users to automatically perform actions on changing data on the detection of specific patterns or conditions.

In the sales context, this helps monitor sales data in Power BI reports and trigger alerts or actions based on real-time changes, ensuring that sales teams can respond quickly to critical events. 

Power BI:

Power BI, integrated within Microsoft Fabric, is a leading Business Intelligence tool that facilitates advanced data visualization and reporting.

For sales teams, it offers interactive dashboards that display key metrics, trends, and performance indicators. This enables a deep analysis of sales data, helping to identify what drives demand and what affects sales performance.

Learn how to use Power BI for data exploration and visualization

Hands-on Practice on Microsoft Fabric:

Let’s get started with sales data analysis by leveraging the power of Microsoft Fabric: 

1. Sample Data

The dataset utilized for this example is the sample sales data (sales.csv). 

2. Create Workspace

To work with data in Fabric, first create a workspace with the Fabric trial enabled. 

• On the home page, select Synapse Data Engineering.
• In the menu bar on the left, select Workspaces.
• Create a new workspace with any name and select a licensing mode. When a new workspace opens, it should be empty.

3. Create Lakehouse

Now, let’s create a lakehouse to store the data.

• In the bottom left corner select Synapse Data Engineering and create a new Lakehouse with any name.

• On the Lake View tab in the pane on the left, create a new subfolder.

4. Create Pipeline

To ingest data, we’ll make use of a Copy Data activity in a pipeline. This will enable us to extract the data from a source and copy it to a file in the already-created lakehouse. 

• On the Home page of Lakehouse, select Get Data and then select New Data Pipeline to create a new data pipeline named Ingest Sales Data. 
• The Copy Data wizard will open automatically, if not select Copy Data > Use Copy Assistant in the pipeline editor page. 
• In the Copy Data wizard, on the Choose a data source page select HTTP in the New sources section.  
• Enter the settings in the connect to data source pane as shown:

• Click Next. Then on the next page select Request method as GET and leave other fields blank. Select Next. 

• When the pipeline starts to run, its status can be monitored in the Output pane. 
• Now, in the created Lakehouse check if the sales.csv file has been copied. 

5. Create Notebook

On the Home page for your lakehouse, in the Open Notebook menu, select New Notebook. 

• In the notebook, configure one of the cells as a Toggle parameter cell and declare a variable for the table name.

• Select Data Wrangler in the notebook ribbon, and then select the data frame that we just created using the data file from the copy data pipeline. Here, we changed the data types of columns and dealt with missing values.  

Data Wrangler generates a descriptive overview of the data frame, allowing you to transform, and process your sales data as required. It is a great tool especially when performing data preprocessing for data science tasks.

• Now, we can save the data as delta tables to use later for sales analytics. Delta tables are schema abstractions for data files that are stored in Delta format.  

• Let’s use SQL operations on this delta table to see if the table is stored. 

6. Run and Schedule Pipeline

Go to the already created pipeline page, add Notebook Activity to the completion of the copy data pipeline, and follow these configurations. So, the table_name parameter will override the default value of the table_name variable in the parameters cell of the notebook.

In the Notebook, select the notebook you just created. 

7. Schedule and Monitor Pipeline

Now, we can schedule the pipeline.  

• On the Home tab of the pipeline editor window, select Schedule and enter the scheduling requirements.

• To keep track of pipeline runs, add the Office Outlook activity after the pipeline.  
• In the settings of activity, authenticate with the sender account (use your account in ‘To’). 
• For the Subject and Body, select the Add dynamic content option to display the pipeline expression builder canvas and add the expressions as follows. (select your activity name in ‘activity ()’)

8. Use Data from Pipeline in PowerBI

• In the lakehouse, click on the delta table just created by the pipeline and create a New Semantic Model.

• As the model is created, the model view opens click on Create New Report.

• This opens another tab of PowerBI, where you can visualize the sales data and create interactive dashboards.

Choose a visual of interest. Right-click it and select Set Alert. Set Alert button in the Power BI toolbar can also be used.  

• Next, define trigger conditions to create a trigger in the following way:

This way, sales professionals can seamlessly use their data across the platform by transforming and storing it in the appropriate format. They can perform analysis, make informed decisions, and set up triggers, allowing them to monitor sales performance and react quickly to any uncertainty.

Conclusion

In conclusion, Microsoft Fabric as a revolutionary all-in-one analytics platform simplifies data management for enterprises. Providing a unified environment eliminates the complexities of handling multiple services just by being a haven where data moves in and out all within the same environment for ease of ingestion, processing, or analysis.

With Microsoft Fabric, businesses can streamline data workflows, from data ingestion to real-time analytics, and can respond quickly to market dynamics.

Want to learn more about Microsoft Fabric? Join our crash course today for a comprehensive understanding!

The relentless tide of data preserves—customer behavior, market trends, and hidden insights—all waiting to be harnessed. Yet, some marketers remain blissfully ignorant, their strategies anchored in the past.

They ignore the call of data analytics, forsaking efficiency, ROI, and informed decisions. Meanwhile, their rivals ride the data-driven wave, steering toward success. The choice is stark: Adapt or fade into obscurity.

In 2024, the landscape of marketing is rapidly evolving, driven by advancements in data-driven marketing and shifts in consumer behavior. Here are some of the latest marketing trends that are shaping the industry:

Impact of AI on Marketing and Latest Trends

1. AI-Powered Intelligence

AI is transforming marketing from automation to providing intelligent, real-time insights. AI-powered tools are being used to analyze customer data, predict behavior, and personalize interactions more effectively.

For example, intelligent chatbots offer real-time support, and predictive analytics anticipate customer needs, making customer experiences more seamless and engaging.

2. Hyper-Personalization

Gone are the days of broad segmentation. Hyper-personalization is taking center stage in 2024, where every customer interaction is tailored to individual preferences.

Advanced AI algorithms dissect behavior patterns, purchase history, and real-time interactions to deliver personalized recommendations and content that resonate deeply with consumers. Personalized marketing campaigns can yield up to 80% higher ROI.

Navigate 5 steps for data-driven marketing to improve ROI

Why Should You Adopt Data-Driven Marketing?

Companies should focus on data-driven marketing for several key reasons, all of which contribute to more effective and efficient marketing strategies. Here are some compelling reasons, supported by real-world examples and statistics:

• Enhanced Customer Clarity

Data-driven marketing provides a high-definition view of customers and target audiences, enabling marketers to truly understand customer preferences and behaviors.

This level of insight allows for the creation of detailed and accurate customer personas, which in turn inform marketing strategies and business objectives. With these insights, marketers can target the right customers with the right messages at precisely the right time.

• Stronger Customer Relationships at Scale

By leveraging data, businesses can offer a personalized experience to a much wider audience. This is particularly important as companies scale. For example, businesses can use data from various platforms, devices, and social channels to tailor their messages and deliver a superb customer experience at scale.

• Identifying Opportunities and Improving Business Processes

Data can help identify significant opportunities that might otherwise go unnoticed. Insights such as pain points in the customer experience or hiccups in the buying journey can pave the way for process enhancements or new solutions.

Additionally, understanding customer preferences and behaviors can lead to more opportunities for upselling and cross-selling.

• Improved ROI and Marketing Efficiency

Data-driven marketing allows for more precise targeting, which can lead to higher conversion rates and better ROI. By understanding what drives customer behavior, marketers can optimize their strategies to focus on the most effective tactics and channels.

This reduces wasted spending and increases the efficiency of marketing efforts.

• Continuous Improvement and Adaptability

A cornerstone of data-driven marketing is the continuous gathering and analysis of data. This ongoing process allows companies to refine their strategies in real-time, replicating successful efforts and eliminating those that are underperforming. This adaptability is crucial in a rapidly changing market environment.

• Competitive Advantage

Companies that leverage data-driven marketing are more likely to gain a competitive edge. For example, research conducted by McKinsey found that data-driven organizations are 23 times more likely to acquire customers, six times more likely to retain them, and 19 times more likely to be profitable.

Real-World Examples

Target: Target used data analytics to identify pregnant customers by analyzing their purchasing patterns. This allowed them to send personalized coupons and marketing messages to expectant mothers, resulting in a significant increase in sales.

Amazon: Amazon uses data analytics to recommend products to customers based on their past purchasing history and browsing behavior, significantly increasing sales and customer satisfaction [12].

Netflix: Netflix personalizes its content offerings by analyzing customer data to recommend TV shows and movies based on viewing history and preferences, helping retain customers and increase subscription revenues.

Data-driven marketing is not just a trend but a necessity in today’s competitive landscape. By leveraging data, companies can make informed decisions, optimize their marketing strategies, and ultimately drive business growth and customer satisfaction.

Top Marketing Analytics Strategies to follow in 2024

Here are some top strategies for marketing analytics that can help businesses refine their marketing efforts, optimize campaigns, and enhance customer experiences:

1. Use Existing Data to Set Goals

Description: Start by leveraging your current data to set clear and achievable marketing goals. This helps clarify what you want to achieve and makes it easier to come up with a plan to get there.

Implementation: Analyze your business’s existing data, figure out what’s lacking, and determine the best strategies for filling those gaps. Collaborate with different departments to build a roadmap for achieving these goals.

2. Put the Right Tools in Place

Description: Using the right tools is crucial for gathering accurate data points and translating them into actionable insights.

Implementation: Invest in a robust CRM focusing on marketing automation and data collection. This helps fill in blind spots and enables marketers to make accurate predictions about future campaigns [5].

3. Personalize Your Campaigns

Description: Personalization is key to engaging customers effectively. Tailor your campaigns based on customer preferences, behaviors, and communication styles.

Implementation: Use data to determine the type of messages, channels, content, and timing that will resonate best with your audience. This includes segmenting and personalizing every step of the sales funnel.

4. Leverage Marketing Automation

Description: Automation tools can significantly streamline data-driven marketing processes, making them more manageable and efficient.

Implementation: Utilize marketing automation to handle workflows, send appropriate messages triggered by customer behavior, and align sales and marketing teams. This increases efficiency and reduces staffing costs.

5. Keep Gathering and Analyzing Data

Description: Continuously growing your data collection is essential for gaining more insights and making better marketing decisions.

Implementation: Expand your data collection through additional channels and improve the clarity of existing data. Constantly strive for more knowledge and refine your strategies based on the new data [9].

6. Constantly Measure and Improve

Description: Monitoring, measuring, and improving marketing efforts is a cornerstone of data-driven marketing.

Implementation: Use analytics to track campaign performance, measure ROI, and refine strategies in real-time. This helps eliminate guesswork and ensures your marketing efforts are backed by solid data.

7. Integrate Data Sources for a Comprehensive View

Description: Combining data from multiple sources provides a more complete picture of customer behavior and preferences.

Implementation: Use website analytics, social media data, and customer data to gain comprehensive insights. This holistic view helps in making more informed marketing decisions.

8. Focus on Data Quality

Description: High-quality data is crucial for accurate analytics and insights.

Implementation: Clean and validate data before analyzing it. Ensure that the data used is accurate and relevant to avoid misleading conclusions.

9. Use Visualizations to Communicate Insights

Description: Visual representations of data make it easier for stakeholders to understand and act on insights.

Implementation: Use charts, graphs, and dashboards to visualize data. This helps in quickly conveying key insights and making informed decisions.

Read more about 10 data visualization tips to improve your content strategy

10. Employ Predictive and Prescriptive Analytics

Description: Go beyond descriptive analytics to predict future trends and prescribe actions.

Implementation: Use predictive models to foresee customer behavior and prescriptive models to recommend the best actions based on data insights. This proactive approach helps in optimizing marketing efforts.

By implementing these strategies, businesses can harness the full potential of marketing analytics to drive growth, improve customer experiences, and achieve better ROI.

Here’s a list of 5 trending AI customer service tools to boost your business

Goals-based approach

Marketing analytics tells you about the most profitable marketing activities of your business. The more effectively you target the right people with the right approach, the greater value you generate for your business.

However, it is not always clear which of your marketing activities are effective at bringing value to your business.  This is where marketing analytics comes in. Running an Amazon seller competitor analysis is crucial to your success in the marketplace. Using a framework to monitor your competitors’ efforts is a great way to ensure you can beat them at their own game.

It guides you to use the data to evaluate your marketing campaign. It helps you identify which of your activities are effective in engaging with your audience, improving user experience, and driving conversions. 

Grow your business with Data Science Dojo 

Data driven marketing is imperative in optimizing your campaigns to generate a net positive value from all your marketing activities in real-time. Without analyzing your marketing data and customer journey, you cannot identify what you are doing right and what you are doing wrong when engaging with potential customers. The 6 features listed below can give you the start you need to get into analyzing and optimizing your marketing strategy using marketing analytics.

 Learn about marketing analytics tools in this blog

1. Impressions 

In digital marketing, impressions are the number of times any piece of your content has been shown on a person’s screen. It can be an ad, a social media post, video etc. However, it is important to remember that impressions do not mean views, a view is an engagement, anytime somebody sees your video that is a view, but an impression would also include anytime they see your video in the recommended videos on YouTube or in their newsfeed on Facebook. The impression will be counted regardless of whether they watch your video or not. 

Learn more about impressions in this video

It is also important to distinguish between impressions and reach. Reach is the number of unique viewers, so for example if the same person views your ad three times, you will have three impressions but a reach of one.  

Impressions and reach are important in understanding how effective your content was at gaining traction. However, these metrics alone are not enough to gauge how effective your digital marketing efforts have been, neither impressions nor reach tell you how many people engaged with your content. So, tracking impressions is important, but it does not specify whether you are reaching the right audience.  

2. Engagement rate 

In social media marketing, engagement rate is an important metric. Engagement is when a user comments, likes, clicks, or otherwise interacts with any of your content. Engagement rate is a metric that measures the amount of engagement of your marketing campaign relative to each of the following: 

• Reach 
• Post 
• Impressions  
• Days
• Views 

Engagement rate by reach is the percentage of people who chose to interact with the content after seeing it. It is calculated by the following formula. Reach is a more accurate measurement than follower count, because not all of your brands followers may see the content while those who do not follow your brand may still be exposed to your content. 

Engagement rate by post is the rate at which followers engage with the content. This metric shows how engaged your followers are with your content. However, this metric does not account for organic reach and as your follower count goes up your engagement by post goes down. 

Engagement rate by Impressions is the rate of engagement relative to the number of impressions. If you are running paid ads for your brand, engagement rate by impressions can be used to gauge your ads effectiveness.  

Average Daily engagement rate tells you how much your followers are engaging with your content daily. This is suitable for specific use cases for instance, when you want to know how much your followers are commenting on your posts or other content. 

Engagement rate by views gives the percentage of people who chose to engage with your video after watching them. This metric however does not use unique views so it may double or triple count views from a single user. 

Learn more about engagement rate in this video

3. Sessions 

Sessions are another especially important metric in marketing campaigns that help you analyze engagement on your website. A session is a set of activities by a user within a certain period. For example, a user spent 10 minutes on your website, loading pages, interacting with your content and completed an interaction. All these activities will be recorded in the same 10-minute session.  

In Google Analytics, you can use sessions to check how much time a user spent on your website (session length), how many times they returned to your website (number of sessions), and what interactions users had with your website. Tracking sessions can help you determine how effective your campaigns were in directing traffic towards your website. 

If you have an E-commerce website another very helpful tool on Google Analytics is behavioral analytics. With behavioral analytics you see what key actions are driving purchases on your website. The sessions report can be accessed under conversions tab on Google Analytics. This report can help you understand user behaviors such as abandon carts. This allows you to target these users with targeted ads or offering incentives to complete their purchase. 

Learn more about sessions in this video

4. Conversion rate 

Once you have engaged your audience the next step in the customers’ journey is conversion. A conversion is when you make the customer or user complete a specific action. This desired action can be anything from a form submission, purchasing a product or subscribing to a service. The conversion rate is the percentage of visitors who completed the desired action.

So, if you have a form on your website and you want to find out what the conversion rate is. You would simply divide the number of form submissions by the number of visitors on that form’s page (Total conversions/total interactions). 

Conversion rate is a very important metric that helps you assess the quality of your leads. While you may generate a large number of leads or visitors, if you cannot get them to perform the desired action you may be targeting the wrong audience. Conversion rate can also help you gauge how effective your conversion strategy is, if you aren’t converting visitors, it might indicate that your campaign needs optimization. 

5. Attribution  

Attribution is a sophisticated model that helps you measure which channels are generating the most sales opportunities or conversions. It helps you assign credit to specific touchpoints on the customers journey and understand which touchpoints are driving conversions the most. But how do you know which touchpoint to attribute to a specific conversion?  Well, that depends on which attribution models you are using. There are four common attribution models. 

First touch attribution models assign all the credit to the first touchpoint that drove the prospect to your website. It focuses on the top of the marketing efforts funnel and tells you what is attracting people to your brand 

Last touch attribution models assign credit to the last touchpoint. It focuses on the last touchpoint the visitor interacted with before they converted. 

Linear attribution model assigns an equal weight to all the touchpoints in the buyer’s journey. 

Time decay attributions is based on how close the touchpoint is to the conversion, where a weighted percentage is assigned to the most recent touchpoints. This can be used when the buying cycle is relatively short. 

What model you use is based on what product or subscription you are selling and what is the length of your buyer cycle. While attribution is very important in identifying the effectiveness of your channels, to get the complete picture you need to look at how each touchpoint drives conversion. 

 Learn more about attribution in this video

6. Customer lifetime value 

Businesses prefer retaining customers over acquiring new ones, and one of the main reasons is that attracting new customers has a cost. The customer acquisition cost is the total cost that you incur as a business acquiring a customer. The customer acquisition cost is calculated by dividing the marketing and sales cost by the number of new customers. 

Learn more about CLV in this video

So, as a business, you must weigh the value of each customer with the associated acquisition cost. This is where the customer lifetime value or CLV comes in. The Customer lifetime value is the total value of your customer to your business during the period of your relationship.

The CLV helps you forecast your revenue as well, the larger the average CLV you have the better your forecasted revenue will be. CLV is calculated by dividing the annual revenue generated from customers by the average retention period (in years).  If your CAC is higher than your CLV, then you are on average losing money on every customer you make.

This presents a huge problem. Metrics like CAC and CLV are very important for driving revenue. They help you identify high-value customers and identify low value customers so you can understand how to serve these customers better. They help you make more informed decisions regarding your marketing effort and build a healthy customer base. 

 Integrate marketing analytics into your business 

Marketing analytics is a vast field. There is no one method that suits the needs of all businesses. Using data to analyze and drive your marketing and sales effort is a continuous effort that you will find yourself constantly improving upon. Furthermore, finding the right metrics to track that have a genuine impact on your business activities is a difficult task.

So, this list is by no means exhaustive, however the features listed here can give you the start you need to analyze and understand what actions are important in driving engagement, conversions and eventually value for your business.  

Develop an understanding of text analytics, text conforming, and special character cleaning. Learn how to make text machine-readable.

Text analytics for machine learning: Part 2

Last week, in part 1 of our text analytics series, we talked about text processing for machine learning. We wrote about how we must transform text into a numeric table, called a term frequency matrix, so that our machine learning algorithms can apply mathematical computations to the text. However, we found that our textual data requires some data cleaning.

In this blog, we will cover the text conforming and special character cleaning parts of text analytics.

Understand how computers read text

The computer sees text differently from humans. Computers cannot see anything other than numbers. Every character (letter) that we see on a computer is actually a numeric representation to a computer, with the mapping between numbers and characters determined by an “encoding table.” The simplest, but most common, is ASCII encoding in text analytics. A small sample ASCII table is shown to the right.

Any human would know that this is just six different spellings for the same word, but to a computer these are six different words. These would spawn six different columns in our term-frequency matrix. This will bloat our already enormous term-frequency matrix, as well as complicate or even prevent useful analysis.

Unify words with the same spelling

To unify the six different “CAFÉ’s”, we can perform two simple global transformations.

Casing: First we must convert all characters to the same casing, uppercase or lowercase. This is a common enough operation. Most programming languages have a built-in function that converts all characters into a string into either lowercase or uppercase. We can choose either global lowercasing or global uppercasing, it does not matter as long as it’s applied globally.

String normalization: Second, we must convert all accented characters to their unaccented variants. This is often called Unicode normalization, since accented and other special characters are usually encoded using the Unicode standard rather than the ASCII standard. Not all programming languages have this feature out of the box, but most have at least one package which will perform this function.

Note that implementations vary, so you should not mix and match Unicode normalization packages. What kind of normalization you do is highly language dependent, as characters which are interchangeable in English may not be in other languages (such as Italian, French, or Vietnamese).

Remove special characters and numbers

The next thing we have to do is remove special characters and numbers. Numbers rarely contain useful meaning. Examples of such irrelevant numbers include footnote numbering and page numbering. Special characters, as discussed in the string normalization section, have a habit of bloating our term-frequency matrix. For instance, representing a quotation mark has been a pain-point since the beginning of computer science.

Unlike a letter, which may only be capital or not capital, quotation marks have many popular representations. A quotation character has three main properties: curly, straight, or angled; left or right; single, double, or triple. Depending on the text analytics encoding used, not all of these may exist.

The table below shows how quoting the word “café” in both straight quote and left-right quotes would look in a UTF-8 table in Arial font.

Avoid over-cleaning

The problem is further complicated by each individual font, operating system, and programming language since implementation of the various encoding standards is not always consistent. A common solution is to simply remove all special characters and numeric digits from the text. However, removing all special characters and numbers can have negative consequences.

There is a thing as too much data cleaning when it comes to text analytics. The more we clean and remove the more “lost in translation” the textual message may become. We may inadvertently strip information or meaning from our messages so that by the time our machine learning algorithm sees the textual data, much or all the relevant information has been stripped away.

For each type of cleaning above, there are situations in which you will want to either skip it altogether or selectively apply it. As in all data science situations, experimentation and good domain knowledge are required to achieve the best results.

When do we want to avoid over-cleaning in your text analytics?

Special characters: The advent of email, social media, and text messaging have given rise to text-based emoticons represented by ASCII special characters.

For example, if you were building a sentiment predictor for text, text-based emoticons like “=)” or “>:(” are very indicative of sentiment because they directly reference happy or sad. Stripping our messages of these emoticons by removing special characters will also strip meaning from our message.

Numbers: Consider the infinitely gridlocked freeway in Washington state, “I-405.” In a sentiment predictor model, anytime someone talks about “I-405,” more likely than not the document should be classified as “negative.” However, by removing numbers and special characters, the word now becomes “I”. Our models will be unable to use this information, which, based on domain knowledge, we would expect to be a strong predictor.

Casing: Even cases can carry useful information sometimes. For instance, the word “trump” may carry a different sentiment than “Trump” with a capital T, representing someone’s last name.

One solution to filter out proper nouns that may contain information is through name entity recognition, where we use a combination of predefined dictionaries and scanning of the surrounding syntax (sometimes called “lexical analysis”). Using this, we can identify people, organizations, and locations.

Next, we’ll talk about stemming and Lemmatization as a way to help computers understand that different versions of words can have the same meaning (ex. run, running, runs).

Learn more

Want to learn more about text analytics? Check out the short video on our curriculum page OR

There are two key schools of thought on good practices for database management: data normalization and standardization. We will learn why does each matter? 

Organizations are investing heavily in technology as artificial intelligence techniques, such as machine learning, continue to gain traction across several industries.

• A Price Water Cooper Survey pointed out that 40% of business executives in 2018 make major decisions at least once every 30 days using data and this is constantly increasing
• A Gartner study states the 40% of enterprise data is either incomplete, inaccurate, or unavailable

As the speed of data entering the business increases with the Internet of Things becoming more mature, the risk of disconnected and siloed data grows if it is poorly managed within the organization. Gartner has suggested that a lack of data quality control costs average businesses up to $14 million per year.

The adage of “garbage in, garbage out” still plagues analytics and decision making and it is fundamental that businesses realize the importance of clean and normalized data before embarking on any such data-driven projects.

When most people talk about organizing data, they think it means getting rid of duplicates from their system which, although important, is only the first step in quality control and there are more advanced methods to truly optimize and streamline your data.

There are two key schools of thought on good practice: data normalization and standardization. Both have their place in data governance and/or preparation strategy.

Why data normalization?

A data normalization strategy takes database management and organizes it into specific tables and columns with the purpose of reducing duplication, avoiding data modification issues, and simplifying queries. All information is stored logically in one central location, reducing the propensity for inconsistent data (sometimes known as a “single source of truth”). In simple terms, it ensures your data looks and reads the same across all records.

In the context of machine learning and data science, it takes the values from the database and where they are numeric columns, changes them into a common scale. For example, imagine you have a table with two columns; one contains values between 0 and 1 and the other contains between 10,000 and 100,000.

The huge differences in scale might cause problems if you attempt to do any analytics or modeling. This strategy will take these two columns by creating a matching scale across all columns whilst maintaining the distribution e.g. 10,000 might become 0 and 100,000 becomes 1 with values in-between being weighted proportionality.

In real-world terms, consider a dataset of credit card information that has two variables, one for the number of credit cards and the second for income. Using these attributes, you might want to create a cluster and find similar applicants.

Both of these variables will be on completely different types of scale (income being much higher) and would therefore likely have a far greater influence on any results or analytics. Normalization removes the risk of this kind of bias.

The main benefits of this strategy in analytical terms are that it allows faster searching and sorting as it is better at creating indexes via smaller, logical tables. Also, in having more tables, there is a better use of segments to control the tangible placement of data store.

There will be fewer nulls and redundant data after modeling any necessary columns and bias/issues with anomalies are greatly reduced by removing the differences in scale.

This concept should not be confused with data standardization, and it is important that both are considered within any strategy.

What is data standardization?

Data standardization takes disparate datasets and puts them on the same scale to allow easy comparison between different types of variables. It uses the average (mean) and the standard deviation of a dataset to achieve a standardized value of a column.

For example, let’s say a store sells $520 worth of chocolate in a day. We know that on average, the store sells $420 per day and has a standard deviation of $50. To standardize the $520 we would do a calculation as follows:

520-420/50 = 100/50 = 2- our standardized value for this day is 2. If the sales were $600, we’d scale in a similar way as 600-420/50 = 180/50 = 3.6.

If all columns are done on a similar basis, we quickly have a great base for analytics that is consistent and allows us to quickly spot correlations.

In summary, data normalization processes ensure that our data is structured logically and scaled proportionally where required, generally on a scale of 0 to 1. It tends to be used where you have predefined assumptions of your model. Data standardization can be used where you are dealing with multiple variables together and need to find correlations and trends via a weighted ratio.

By ensuring you have normalized data, the likelihood of success in your machine learning and data science projects vastly improves. It is vital that organizations invest as much in ensuring the quality of their data as they do in the analytical and scientific models that are created by it. Preparation is everything in a successful data strategy and that’s what we mainly teach in our data science bootcamp courses.