Data Quality and Retension

Welcome to the module Data Quality and Retention. In this module, we will start by learning about data quality, its importance, and how to define storage policies that improve the quality of your data insertion and retrieval. Then we will take a look at the backups and the backups policies and also expand on the topic of cold storage and how to use the cold storage to store backups of your data. Then, we will learn about data cleanups and how to remove the data to protect the user privacy.

Data Quality Overview

First, let's learn about data quality. The quality of the data is unique in that regardless of your industry, you would want to get a good quality of data to run your business. Data quality frameworks start from the data injection process on how you collect the data and what the user provides. The storage part makes it possible to store the data that is consistent and lacks duplication and missing values. On the other hand, data quality frameworks ensure that you are providing the data in a correct manner that helps build reports and summaries. Rich content and the data gives a lot of value to the stakeholders in driving the business. And if the data quality is high, that allows stakeholders to stay ahead of the competition. Data quality frameworks allow your data platform to avoid data corruption by handling the missing or duplicate or invalid inputs. The frameworks also assist in keeping the sensitive data protected and safe. During the data injection process and during the update process, we can build and update the metadata for all the data that lives in our platform. The data quality frameworks are defined by the business goals, and your stakeholders share the requirements for the data. For example, in some industries, you might want to remove the records that have missing values, while in others, you might just simply normalize the records and store them. In the framework, you use the procedures defined by compliance to keep the data protected and regulated. As a fact, keeping your data consistent allows your data platform to be performant.

Data Quality Framework

Let's explore what's important for the data quality framework development. The data quality framework enables data engineers to clean up corrupt data, to ensure lack of duplicate records, and to make the data and entities consistent. Throughout the pipeline, the data quality framework ensures that the compliance procedures are implemented and executed. Also, this is more important for the data engineers to continuously improve the framework and identify any gaps in the quality assessment for the data injection. As mentioned, the data quality standards are based on the industry that your organization belongs to. How strict, how verbose, and how much sensitive these standards are is based on how sensitive the data you are handling and the scope of the business such as are you working only in one municipality or are you a multinational corporate? The standards also define policies on how you should transform the data as the data continues to live in the platform, ultimately anonymizing the data to make sure that the user privacy is respected, but your organization has a historic view of the business. The pipeline that your data engineers develop are responsible for handling the data inputs, assigning any validations to the data, processing how to ensure the data that is being added is consistent by removing duplicates, and adding any missing values or normalizing it. The pipeline should generate the metadata as a byproduct to help other processes and data engineers or your support teams to understand what data was collected, where it is stored, what are the regulations, and compliance rules and how to sum up the data for stakeholders to use it. We also discuss that the data quality framework uses continuous learning to optimize the process. One way is to assign a score on the quality of the data. For example, the more corrupt the data, the less score for the quality of the data. You can use methods such as sanitization, sampling, or actively rejecting the invalid or duplicate data to make sure that the data is consistent and valuable. You can label the data to help regulators and internal auditors to make sense of the data and to perform operations and procedures to ensure data security. Remember, the data quality frameworks depend on your industry. That is why you have to work with compliance and regulatory requirements to assist your data engineering teams to build a scalable data platform.

Data Storage and access policies

The data storage engineering allows the platform to be scalable. However, the storage has to comply with two important features, privacy of the user data to ensure no one accesses the data when it is not necessary and to secure the data. Not just in the data centers, but also in transit, meaning using the industry leading security practices for networks and the connectivity. The underlying storage platform that you choose defines how available your system will be, for example when there is a lot of load on the system. The platform also defines how encryption would work, for example data encryption at rest and data encryption in transit. Next, you can also define the access control policies such as creating user accounts or tokens to gain access to the data. Always keep your data storage platform up to date and patched for all the latest security updates to prevent any vulnerabilities. We mentioned user accounts or tokens. These can be created separately for the engineers or your internal and partner applications to gain insights to the data or for your stakeholders to access the user data or the dashboards to preview the business. These accounts can also be created and used by end users such as your customers to use your solutions. When creating the user accounts, ensure that you are using the industry‑leading practices such as creating the accounts with limited bare minimum permissions and scopes to minimize the attack vector. If a permission is not needed, do not grant it. The authentication flows must also be secure and better tested. For example, in finance and payments industry, you should use multifactor authentications such as 3DS2 protocols. Not just the user data, but any tokens or secrets must also be encrypted in your data centers to prevent misuse and generate the tokens for as short time as possible to prevent abuse. Note that the data access must be given for compliance reasons. For example, a user must be able to see what data you have for their profile, and the access should allow the users to update their details or request complete deletion of their data or the features to allow them to download their data. Even when allowing the users to take control of their data, you should build the systems that prevent mistakes from being made by the end users to cause damage to their own data. The user's data is always the most important part of your business. Always protect it to maintain the trust between yourself and your customers.

Backup and cold storage policies

Keeping the data for long term also needs to be compliant. When building the backups of the data stored in your data centers, ensure that you are keeping the data privacy in mind. For example, if a user requests their data deletion, the backup copies of the data must also be removed. The metadata of the backups can help identify what data is stored in which backup, how to remove it, and how to maintain consistency of the data after each operation. One of the best ways to keep the data stored with lower costs for the organization is to use cold storage backup options. The multi‑temperature storage options are hot storage or the hot data in some contexts, which is a term used to indicate the data that is most frequently accessed, notably the live environments. Then, we have the warm storage, the data that is not frequently accessed but still needs to be readily available for the customers. Lastly, we have cold storage or cold data, which is the data kept for backup purposes and is accessed infrequently or not at all, but it is regularly checked for validation and data consistency. Low‑cost hard drives can be used to keep the data in backups with regular consistency and data corruption checks to ensure data safety. The data is kept in multiple backups because cold storage is not frequently accessed, so the data loss would be discovered too late. With each change, each review, build a proper auditing report to share any breaches in the data with your customers. As needed, always keep the data clean and minimal. During the data deletion from the production, ensure that the data is removed from the cold storage as well. Or as an alternate, anonymize the data before storing in the cold storage backup options. Data validation is a huge topic. But to get started, we can perform various methods of integrity validations such as using the data checksums to validate the data regularly and using data scrubbing approach to ensure the data is valid and can be read properly. While this helps, it is even better to use properly designed software that is suitable to handle long‑term data storage such as ZFS file system that has built‑in support for snapshots and validation checks. Human error is always part of the big picture. Keeping multiple copies of the data can help minimize the human error, but also ensure that if there is a hardware failure, you can get the data from another storage location. Your IT teams must always label the cold storage with the metadata that indicate any regulatory procedures required for that data.

Data removal

While not the last step in the data management pipeline, data removal is crucial in maintaining compliance with global laws. The data removal entails that when your users request a data deletion for their profiles or when your application no longer needs their data, you delete their data permanently and not just soft delete it to keep it stored in your data centers. This is not just an ethical call but a requirement by the regulation. Often, each regional regulation offers a set number of days in which the organization must delete the user data after they have received a request. Typically, that is 90 days. Sometimes when it is not possible to completely delete the data for business or reporting reasons, regulations allow to anonymize the data to ensure the person cannot be identified by the data. To ensure all the data is removed, that metadata plays a very crucial role. Metadata provides clarity on what data lives where and how to handle the deletion. Metadata procedures must be built to provide context on where the data lives if these copies are in backups, and metadata must be able to point to the right backups and allow for deletion. This is why it is recommended to always include the regulatory procedures in the metadata to assist the IT and data engineering teams with proper data removal steps and the tooling. Lastly, remember that when you are done with the data of the users, properly and permanently remove the data from your systems and give users the ability to have an insight on this procedure.

Summary

That brings us to the wrap of this module. In this module, we took a closer look at the data quality and how we can define and build data quality frameworks. Then we looked at how to use storage technologies such as databases and which other features are needed for better security such as encryption, data access policies, and we also discussed that we should only grant minimum permissions to the stakeholders and users. Then we learned how to use backups to keep our platform available always and to minimize the cost using cold storage backup options. Lastly, we learned how to remove the user data, not just from the live platforms, but also from any backups to ensure compliance with global regulations. And this is the wrap for this course. Now you have the knowledge and understanding of the data management framework in the data governance procedures. To continue learning, the exercise files of this course contain various resources that can help you learn more on the topic and take a practical look at how big enterprises are doing their data management pipelines. Thank you for joining the course. And if you have any questions, you can leave a comment on the course.