Data Quality

Key Concepts, Principles, and Terminology Related to Data Quality

Be it inventory management, marketing trends and strategies, understanding customer preferences, personalizing the customer experience, optimizing the supply chain, fraud detection and security, you'll find the role of data everywhere. Data is one of the most valuable asset that any organization has. What are the key concepts and terminology related to data quality and data governance? Let's discuss them in this lesson. Let's start by defining what data quality is and why it matters. Data quality is essential for any organization that wants to use its data effectively. Data quality pertains to the accuracy, reliability, and relevance of data in a dataset, all of which have a deep impact on decision‑making and success for the business. No matter how much data you have, its quality and its effective views matter the most. For example, in online retail, accurate sales analytics data ensures targeted marketing and inventory optimization. Another example is healthcare where perfect electronic health records are essential for accurate diagnosis and patient safety. Poor data quality in any such scenario can be disastrous for decision‑making and service delivery. It's all about ensuring the data is fit and accurate for its intended purposes and is logically correct and consistent so that it meets the specific needs of the people who will be using it. And when it comes to data quality, it's not the responsibility of a single person or just any particular department. You cannot just hire a dedicated person to maintain data quality for all the data in transit or stored and be relaxed. Maintaining data quality is an enterprise‑wide responsibility and everyone who interacts with data directly or indirectly. What that means is it's a shared responsibility for everyone at different levels responsible for it throughout its lifecycle. When we talk about the various roles to maintain data quality, we have roles such as the famous data stewards, data analyst, data governance council, and business users. We will discuss more about roles and responsibilities in detail in our upcoming lessons, but this was just to introduce you to different terminologies so you could relate to them as we progress.

Data Quality Dimensions

Let's dive deeper into the specifics now. Let's discuss the various aspects or dimensions of data quality to ensure your data is truly reliable and accurate. Data quality dimensions are a set of criteria that can be used to assess and improve the quality of data so that organizations can make better decisions based on that. The six most common quality of the data dimensions are accuracy, completeness of data, consistency of the data, timeliness, validity, and uniqueness of the data. Let's discuss each of them one by one. Let's start with the data accuracy. Is the information in your dataset free from errors or mistakes? For example, in a medical database, the accuracy dimension ensures that patient records, including diagnosis and treatment are error‑free. A simple mistake could lead to catastrophic consequences. This quality dimension focuses on the correctness of data. Another data dimension is the completeness of data, which assesses whether your data contains all the expected and required elements. For instance, if you're analyzing sales data, are all the sales transactions included in your dataset or is there any gap? Missing data can lead to incomplete insights and decisions. Moving to data consistency, high quality data is uniform in its format and content. Inconsistent data, on another hand, could lead to confusion and misinterpretation. For instance, if you're dealing with dates in various formats like DD‑MM‑YYYY, MM‑DD‑YYYY or maybe YYYY‑MM‑DD within the same dataset, it can cause errors in date‑based analysis. Another important data dimension is its timeliness. Is your data up to date or are you using outdated information? Like in financial analysis, using outdated stock prices can result in inaccurate predictions and investments, or an old medical diagnostic report may not be helpful at all without the latest ones. Timeliness is all about the relevance of data. Let's talk about another important data dimension, which is validity. This dimension ensures that your data confirms to define rules and constraints. For example, in a database of customer information, all email addresses should adhere to a specific format like Username@domain.com. An email with just a username is meaningless without the domain. Last, but not least, is uniqueness. Data is considered unique if each of its record is unique without any duplicate records within a dataset. Imagine if a customer database contains two records with the same customer ID, the data is not unique. The name could be common with multiple customers, but the customer ID must remain unique for each customer. To put these six data qualities in perspective, let's take the example of a retail company, Globomantics, a dynamic automobile startup, specializing in electric scooters. Data quality in this context means that product description are free from errors. All sales transactions are recorded. Scooter models and categories are consistently formatted. The inventory levels are up to date. Customer information follows a valid format, and only unique products with a unique chassis number for every electrical scooter are considered for analysis. A question for you, which of these six data quality dimensions do you think is the most important to ensure high data quality and why? Do you recall a situation where a lack of data quality had a negative and embarrassing impact?

Impact of Poor Data Quality

Poor data quality can have a devastating impact on your business from lost revenue to damaged customer relationships. The impact of poor data quality has led to several famous and costly mistakes. For instance, the 2008 world financial meltdown was fueled by bad data that overstated the value of mortgage‑backed securities and other derivatives, leading to widespread financial collapse, evictions, and job losses in many countries. In 2017, an airline from the USA overbooked a flight from Chicago to Louisville and forcibly removed a passenger from the plane. The incident was caught on video and went viral, sparking widespread outrage and costing an estimated $1 billion US in lost revenue and brand damage to the airline. In 2015, a bank lost$2.8 billion US due to trading errors caused by poor data quality. The bank's traders were using outdated data and models to make trading decisions, which led to this significant loss. Additionally, poor data quality has led to an average loss of $15 million per year globally for different organizations, causing morale issues and lost opportunities. Poor data quality can have several negative impacts on businesses. Firstly, it can lead to a loss of credibility for the customers. Secondly, inaccurate data can result in bad decision‑making, huge revenue loss, and can also damage customer relationships. Try to recall the costliest mistake made as a result of poor data quality that you personally experienced. How did that affect the outcomes and what steps were taken to address that?

Data Governance Roles and Responsibilities

Let's shift our focus to the central and vital topic of data governance. Let's see who does what in data governance. It is all about defining roles and responsibilities to ensure that data is managed effectively and consistently. Let's start with the data steward's role. They are the go‑to person for data‑related queries in their designated business units or data domains, responsible for the day‑to‑day management of specific data assets. They ensure and maintain data quality, and the data is compliant with data policies and standards. They categorize and classify data, implement data policies, and maintain data integrity. In a large organization, business units like sales, human resources, and finance may have various data domains within it such as orders, employees, and payroll respectively. Each data domain is managed by one or more data stewards, responsible for maintaining accuracy, completeness, and consistency, managing security and access, ensuring compliance, maintaining documentation, and resolving data‑related issues. For example, a data steward's typical work schedule will include meeting a data owner to discuss data quality issues at the beginning of the day, then implementing data quality policies and procedures, monitoring data quality and identifying data quality issues, collaborating with others responsible for resolving data quality issues, training employees on data quality, meeting with data governance council to discuss data quality initiatives, and at the end of the day, documenting data quality processes and procedures. Data stewards play a crucial role and are involved with everyone everywhere. They have a more hands‑on approach and work closely with the actual data. Then you have the data owners. They have ultimate accountability for specific datasets or data domains. They make decisions about access, usage, and retention policies for their datasets. They are responsible for overall quality, management, data security, and compliance of specific data assets. For instance, the data owner for a company's financial data might be the chief financial officer, CFO. The CFO would be responsible for making decisions about the access, usage, and retention of company's financial data. They would also be responsible for ensuring that the data complies with all applicable financial regulations. After data stewards, data owners, let's discuss the data governance council. It is a high‑level committee that serves as the central hub, responsible for developing, supervising or overseeing data governance policies and practices. It includes experts and members from various departments who understand the business and data intricacies and provide a broader and strategic approach to set the policies and direction for data governance across the organization. Try to visualize that a data governance council for a healthcare organization might include representatives from the clinical operations, information technology, finance, and legal departments. Technology roles such as database administrators, software developers, and cybersecurity staff are the roles responsible for the technical aspect of data management, including data creation, update, deletion, backup, archiving, and ensuring data security. For example, a database administrator or a software developer often manages data‑related processes while cybersecurity staff focuses on protecting data from threats. Or let's put it this way, a data administrator for a bank might be responsible for managing the bank's customer database while the cybersecurity staff would be responsible for ensuring that the database is secure and that the data is accessible only to the bank's employees and its customers. Now let's talk about the auditors that are internal or external. Not liked by many, they are the ones who are simply trying to ensure that the data management practices are aligned with organizational policies and compliance requirements. They play a vital role in assessing and validating data governance processes. Imagine a situation, an internal auditor for a financial service company might be responsible for auditing the company's data governance practices to ensure that they comply with all applicable regulations. An external auditor might be hired by the company to perform an independent audit of its data governance practices. In addition to these core roles, there are a number of other individuals and teams that play an important role in data governance. For example, business owners, data users, front line and back office staff, and management all have a responsibility to follow data governance policies and procedures. All the roles collectively work together to ensure that the data is managed effectively, is of high quality, and complies with regulation and organizational policies. Data governance is a collaborative effort that involves individuals with different responsibilities to maintain and leverage data asset for informed decision‑making and business success.

Best Practices for Ensuring Data Quality

You might have heard this statement already, that in today's data‑driven world, data is gold, and the quality of your data is extremely important. However, maintaining data quality becomes a complex challenge, especially as data volumes continue to grow exponentially. To address this challenge, organizations need to adopt a comprehensive approach to data quality management and implement best practices that cover the entire data lifecycle from data creation, collection to data analysis and decision‑making. So, what are the best practices that enhances the quality of your data and how can you apply them to your data? Let's discuss some of the best practices for ensuring data quality. Let's start with data profiling. Data profiling is a technique for analyzing data to discover its characteristics and patterns and detecting anomalies such as duplication, lack of consistency, and lack of accuracy and completeness. By understanding the characteristic of your data, you can identify areas that require improvement and ensure that your data aligns with your expectations. Basically, it helps you to understand your data in determining what data is of high quality and meaning. Data profiling has several use cases such as query optimization, data integration, scientific data management, data analytics, project management, and data discovery. Moving further, you have data validation, preventing errors at its source, which is the process of checking data as it entered or imported to ensure its accuracy and completeness. This can be done manually or through automated rules. Data validation prevents errors from entering your system in the first place in the beginning itself, saving you time and effort in the long run. Another important thing is data cleansing. After you have analyzed and validated the data, cleaning your data becomes obvious. Correcting any error or removing any duplicate records regularly is essential for maintaining data quality. This process can become more complex when dealing with entities like names and addresses, but it's crucial to resolve and deduplicate records effectively. For example, in a customer's database, data cleansing can involve merging duplicate customer records and updating outdated contact information. Now let's talk about data completeness. Missing data can significantly impact the accuracy of your analysis and decision‑making, so any incomplete data must be handled carefully. The business must decide whether to eliminate a record altogether, which has missing information, accept them with proper annotation or with certain assumptions or seek additional information from its data sources. Additionally, maintaining clear and comprehensive documentation for your data, including data dictionaries, metadata, and data lineage is also an important matter to ensure high data quality. Lineage tracking for data sources and results help you understand the origin and transformation of your data, making it easier to identify and resolve issues. Data documentation is often overlooked, but it plays a critical role in maintaining data quality. Then we have data quality monitoring, which is a systematic process and not just a one‑time event that involves the continuous observation and evaluation of data to ensure its accuracy, completeness, consistency, and other quality dimensions. This monitoring is crucial because data is dynamic and subject to change, and ensuring its ongoing quality is essential for making reliable decisions. This can also be done through automated tools that identify anomalies and trends in data quality. For example, in a financial institution where a vast amount of transactional data are generated regularly, data quality monitoring plays a pivotal role. Furthermore, we must establish and enforce data governance policies for effective data management and high data quality by defining data ownership, access permissions, and data standards. For example, in a healthcare organization, data governance policies ensure that patient data is handled with care, safeguard patient information, ensure compliance with regulations, maintain the highest standard of data integrity and security, and comply with legal requirements. All right, so in a nutshell, the best practices for ensuring data quality include data profiling, data validation, data cleansing, data completeness, data documentation, and data quality monitoring throughout the data lifecycle when the data was created, when it was used, and finally, when it was removed.

Summary

Okay, great. So let's wrap up what we learned so far. In this module, we discussed some of the crucial concepts like the impact of poor data quality on customer relationships or loss of revenue, data quality dimensions such as accuracy, completeness or timeliness, roles and responsibilities such as data stewards, data owners, and the data governance council. Now imagine if you were to play a role in the data governance structure somewhere, which role do you find most intriguing or challenging and why? How do you think effective data governance can positively impact your organization or project? This is it for this module. See you in the next one, discussing data normalization.