Data runs the enterprises and enables businesses and stakeholders to make decisions that help them drive success

Data management, working closely with the data governance processes and protocols, helps implement regulatory processes to safeguard the user data.

We will learn, how to implement industry standards to manage the user‑generated data and the data owned by your enterprise. In this module, we will take a look at data management and what it entails.

• We will focus on the compliance and the regulatory aspect of the data management.
• Then, we will learn how to store the data and how to handle the data that is provided by the users.
• Next, we will take a look at the storage for the data and how it helps us structure our data platform efficiently.
• With that, we will explore how to gain control over the data and the build processes that assist in auditing our internal procedures and ensuring that we remain compliant with all our applicable regulations.

Data Management Overview

Data management entails the data platform and the compliance procedures. The main purpose of the data management process is to ensure that the data is collected in an efficient manner, especially when the data is coming at a high velocity, and then storing the data in big data platforms to build warehousing solutions and business intelligence reports. The data management frameworks support data processing to parse the data properly, check for the data quality, remove the duplicate or incorrect data, and sometimes attempt to correct the data by filling in some missing fields and also to apply the retention policies on storing the data for longer periods of time in our own data centers. Using the data management procedures, we build the data platforms that have higher accuracy and provide consistent reports to the business stakeholders to make data‑driven decisions. Data management allows your IT teams to apply industry‑standard authentication protocols to control the access to the data and only provide the access that is needed by the said party to function and to be able to do their job, nothing more and nothing less. Because of the sophisticated nature of the procedure, data management is an effort driven and executed by the technical staff such as your data engineers, infrastructure engineers, and software engineers under strict guidance and supervision of the risk officers. The risk officers, such as the data protection officers under the GDPR regulation, must be involved to guide your engineers on building the platforms that adhere to and comply with the required set of privacy and security requirements as are applicable to your organization. Lastly, but the most important element is the end user that is using your products and generating the data that fuels the entire data platform.

Compliance and Regulating Data

The regulations play a very important part in the data management pipeline development. Almost all the data that your organization generates and gathers from the user is sensitive in nature in one way or another. As a regulated organization, it is your responsibility to ensure that the data is protected from all bad actors and external or internal threats. This includes the data that does or can identify a real living person. The same regulations apply to the data that is stored in the backups. Regardless of the current accessibility of the data, the sensitive nature of the information must always be protected while the users should have visibility on the data that your organization stores. Often, the backups are retained. And even though the data is removed from the live databases, the backups might contain a copy of the user's personal data. The data pipeline must have the visibility on the entire data catalog. The regulations are very strict about personally identifiable information. In all the regions across the globe, the terminology might be different, but the data and the strictness is the same. The personally identifiable information is the data that belongs to the user, such as their name or their address, but not the version of software that they use to interact with your application. The data represents a living identity in the real world, a person. The data can be captured in any manner during the account creation such as their email, password, authentication information, or it could be captured during a required procedure on your website such as a payment method for checkout, and this could include but is not limited to the credit card numbers and more. All of this information is very sensitive and could cause damage to a person or their properties and thus is highly and strictly regulated by the governments across the globe. While the underlying concept is the same, the personally identifiable information is usually referred to as personal data in the European regions. You, as the data processor, have a responsibility to ensure the data is safe, protected, and is never abused. The personal data is the data of the users, not limited to this list but including the name, the date of birth, and also their addresses and their ID numbers such as social security numbers, citizenship numbers, and more. Moreover, the email addresses and phone numbers are also a type of linked personal data. They are called linked personal data because they can directly link to a person and can identify a living person in the world. Then, we also have linkable data types such as gender of the person because it does not pinpoint exactly who the real user is. Similarly, the race, religion, and other identities are linkable types of data because they can identify a group of people. Then, the non personal data includes the personal data that is masked and abbreviated, such as instead of the full name just the initials of a person. And similarly, instead of showing one of records of a user account, you can build summarized reports of all the users who have interacted with your solutions. Last but not least, anonymity of the users. Even if you have to present the data, removing the email addresses or the phone numbers and generating random identifiers could help maintain the user privacy. Let's see this in action. This user data table for the employees contains the information such as ID, the first name, last name. This is personal information that can link to a real person. The birthday and email, personal or business email are also linked to the person. However, the location, the position, or gender is not directly linked. For the sake of simplicity, the table is very short and could not form enough groups to demonstrate. But these values are less sensitive in nature and are to be processed and stored with different rules and regulations, while the user's sensitive personal information must be stored with more vigilance. Because of the sensitivity of the data, the compliance and risk officers must work together with the data platform engineers to build the systems that ensure that data privacy is respected.

Data Modeling and Data Storage

Data modeling and storage policies are an important part of the data management pipeline. For data modeling, data engineers and DBAs working with the compliance and risk departments and the business stakeholders define the structure of the database partitions, the table, the database schema, and any indexes for performance. The data platform maintains the data generated by the users in a database schema often backed up in a cloud platform and controlled by visuals and reports for the metadata that helps the stakeholders understand where the data is stored and how to access and handle it. An efficient data modeling exercise with high‑quality database schema allows for a performance solution. For example, the database that your enterprise uses might be relational for the purpose of user data storage or a document store for some other content. Whichever choice you make, you need to include regulations and compliance in the schema, keeping the personally identifiable information separate from the masked and anonymous data that is less sensitive in nature. The backups are also part of the data modeling, and the backups can live in the same data platform and warehouse along with the live data, or you can have a dedicated cold storage platform to retain data for longer periods of time. During the modeling, you can also define how long the data should remain in the live environment or the backup. And when the data is no longer needed by your organization or the application, define the procedures that remove the data from the data platform. This way, the user's privacy is not abused and their data is never misused by any bad actors. With data modeling, the data storage frameworks help in ensuring that the data is always available. Regardless of your database schema and the number of tables, the platform must be able to cope with the incoming data, the velocity and the size of the data. A typical example of this cluster is big data, which is beyond the scope of this course. The storage options should have replication to avoid disaster and data loss. Compliance defines the rules about the backups, data storage policies, how to process, how to share the data, and how to make the data available for your customers and for your stakeholders. Important thing to remember, while the data model is built and defined in the software by the DBAs and the data engineers, they are guided and regulated by the risk officers of an organization and are used by the stakeholders to run the business.

Metadata Management

Metadata helps bring visibility to the properties of the data. The metadata is also called data about the data, meaning the additional properties and details that are stored with or alongside the data. The metadata of the data allows an organization to understand what retention policies have been defined for a set of data. It helps the data engineers and DBAs define access controls and permissions and also various regulated processes. The metadata of the data can live in the same warehouse or the data platform, or perhaps it can be stored in a dedicated warehouse, which can be created for the metadata purposes only. However, metadata must stay within the bounds of regulation and must not expose any sensitive information about the users. For example, if you are storing metadata about the payment methods of a user, then you need to make sure that the credit card or other payment information is not stored in the metadata. The metadata is the property of the enterprise provided that it stores only the data that helps the company run the regulatory processes to keep the data secure, private, and out of bad actors' hands. The purpose of metadata is bigger in the entire data platform. The metadata helps assign the regulation process to the data at various levels such as at the database level, table level, or at a user or the specific piece of information level. Metadata can also contain how the data can be accessed, by whom it can be accessed, and what other industry standards are needed to be met. For example, does a requester need to have two‑factor authentication enabled before accessing the data, or should a user be located in a specific country or region? Apart from the regulatory requirement, metadata can make it easy for the data engineers and the DBAs to find the data in the data warehouse to remove, modify, mask, or move to a new place. The metadata is like a guiding map that helps the organization apply and run various procedures. For example, metadata can help your IT teams define a policy to delete certain data after 3 months. Of course, this depends on your industry and the compliance rules that your organization needs to meet. Then, you can add this information to the metadata for that table or database. Similarly, metadata can be used to define the recipients of notifications and alerts in case a breach is detected or when a change in the data has happened. This way, you can define who has the visibility on the data. With these characteristics, metadata is a powerful tool to implement data governance and ensure your business remains compliant. Metadata tools can also assist access control from within the database and from beyond the cluster. Metadata can assist in defining the schema that is more compliant. For example, in payments and financial technology, you cannot store the entire transaction information in a single record in a database. You need to separate the information based on its sensitivity and then store it. This information can help define a schema that works best for regulatory requirements. And for this, metadata is often defined in collaboration with the risk officers of your organization. Various industries, regions, and business types require that an organization have a risk officer such as a data protection officer for GDPR. Remember, your metadata definition in the data management pipeline helps in the long term to quickly find the information in the data cluster and assist in applying various regulatory procedures to the data.

Data Access and Audit

A data platform must define access controls and permissions to allow for proper audits of the platform. While you need to ensure that the data is kept safe, you do have to offer the data for access and processing to various actors such as stakeholders, data engineers, the end users, or your partners if the compliance allows and in a regulated manner. For example, the stakeholders can use reports generated by the data to make data‑driven decisions for the success of business. The engineers who build the data platform and the solutions on top of these could also use the data, for example, in order to provide support to the customers. Then, often applications also provide the data to the customers such as they enable user to user communication or to build social platforms. Then you would need to offer some access to the user data. In scenarios where your organization does not offer social features, the dashboard could offer the user features to review what data we, as an organization, have on their profile and also offer them access to remove the data as they like. Note that all of these data access pipelines need to have regulations and compliance in place. You cannot show or grant access to any user data without their permission. How to request permission depends on your industry, the process with which the customer is requesting, and how sensitive the data is. The access to this data can be controlled by defining strict policies against each operation that is performed on the data. The security department of your organization helps in defining these policies to ensure safety. The policies must define access that are controlled at an entity or operation level, thus enabling the platform to allow every operation that is needed for the business but reject every other operation. The data platform should also include abstraction layers that prevent malicious users from sniffing the internal details of your data platform, such as the database names, table schema, or the keys that are used. If possible, grant minimum permissions and never grant write permissions to anyone. Auditing process is mandatory and is required by the law and regulation. For example, GDPR in Europe and CCPA in California require proper auditing reports and details on how the data is processed. In some industries, such as finance, health care, or government, you develop an audit plan and have a team of internal auditors assist your business stakeholders and executives to drive the business in a regulated manner. For each operation, it is easy to create a list of logs as the activities happen. This detailed documentation not only helps your organization stay on top of everything as it happens, but also allows you to provide reporting to the regulators. Communication with the regulators and the customers is key. Using the communication, you can demonstrate the responsibility and build trust with your customers. Auditing is a big part of data governance. Auditing with compliance makes you responsible and trusted by your customers.

Summary

This wraps up the module. We learned what is data management and took a look at how compliance dictates and guides the entire process. Data modeling is a process in which data engineers, DBAs, work closely with security, stakeholders, and risk officers to build data models and use storage policies to implement compliance and availability. We then learned what is metadata and how to use metadata to explain the data, where it is captured from, how long should it be stored, and what other processes and procedures to take into account when removing it? Lastly, we looked at access control policies and how to audit the activities and operations. Now in the next module, we will learn about building a data quality framework and how to retain the data and define data removal procedures.