Data Quality | Tutorial | Data Quality Management | Data Quality Assessment | Data Quality Issues - YouTube

Hello, and welcome to this data quality tutorial. First, let us understand. What is data quality?

Now, basically data is of high quality if it does not suffer from data issues.

There are many possible issues with data. We will see multiple examples shortly.

Now, data is used for many functions such as operations, customer management, marketing analysis and decision making.

if the data is of poor quality, there are several problems like wastage of time and money due to incorrect reports, poor decisions and

frustration.

Now, let us see examples of data issues.

First issue is incompleteness. Here is the Customer table

with columns FirstName, LastName, BillingAddres, Shipping Address and Email.

For example, Dave has a billing address of 111 Main City street, and the email is [email protected]

Now, here LastName is missing, Shipping Address is missing and the Email is truncated.

Robert has the default BillingAddress and

his ShippingAddress is missing.

It is also possible that the Customer table misses records that exist in the source data, or the customer table is not updated.

Second data quality issues is accuracy. Here is the employee table with columns, Name, Position, Department and Email.

Now, Anna's Position and Department is interchanged.

Berry's Position has a typo in Software.

Charles' Department has a junk value.

The third data quality issue is inconsistency. Here is the Order table with columns, OrderNumber, CustomerId,

ShipDate and Total.

Now, the totals are in different currency. This one is in dollars, and this one is in euros.

The currency should be consistent.

Fourth data quality issue is Invalidity.

Here is the Product table with columns, ProductCode, Name, UnitsAvailable and RetireDate.

Now, UnitsAvailable is negative in this row. Which is impossible.

Also Widget2 is already retired, but still present in this table.

The fifth data quality issue is Redundancy. Here is a Customer table with columns, FirstName, LastName and Email.

Now, the second row is a duplicate of the first row because the email addresses are the same.

The sixth data quality issue is Non-standard. Here is the Order table with columns, OrderNumber, CustomerID and Total.

In the

second row, the total has four places after the decimal point, so it is in non-standard format.

Now, let us see the process of data quality assurance, which can be used by the data quality analyst.

It has three main activities.

First is data profiling.

It involves exploring the data

to understand and analyze it.

Then identify

the data quality issues and then summarize the data in Table Row counts, blank values, duplicates and so on.

If the data quality issue is widespread, the data quality is poorer.

For example, if the Customer table has 2% rows missing, the data quality is higher than if the Customer table has 20% rows missing.

Data quality can be expressed in data quality metrics such as % Complete, % Accurate, % Conformance.

(that means conformance to business rules

like value >= the lower range).

Now, manual data profiling with SQL queries is effort intensive, so only a small subset of data can be profiled.

However tools like SQL Server Data Quality Services do not need writing sql queries and

can run checks on the entire data set quickly.

The next activity in data QA is Data Cleansing. It involves fixing data issues.

Simple updates fix data quality issues like misspellings, typos, incorrect formats and missing foreign keys.

In case of duplicates,

one of the rows is selected as the Master.

Data in the duplicate rows is used to enrich the Master or just discarded.

Selective imports bring in the missing rows.

Obsolete data is deleted.

The important point is that due to the risk associated with data changes, such changes are NOT applied to the database directly.

Each proposed change is listed and

reviewed by a domain expert also known as the Data Steward.

The Data Steward approves or rejects proposed changes. Only the approved changes are applied by the Data Quality Analyst.

The next activity in data QA is data monitoring. The data should be maintained in a clean state.

Users entering new data check business rules first.

Users who consume data, submit data quality issues for review.

The Data Quality Analyst

runs checkers like parsers, spell-checkers, business rules checkers, duplicate checkers etc. to find out

data quality issues and submit them to the Data Steward.

If the Data Seaward lacks in domain knowledge, he or she can use

pre-built knowledge bases with business rules from multiple domains like address, zip code etc.

That's all in this video. Thank you and see you in my other videos on Database normalization.