DEV Community

Cover image for Data Governance Hands On with Amazon DataZone
Neylson Crepalde for AWS Community Builders

Posted on

Data Governance Hands On with Amazon DataZone

At re:Invent 2022, AWS announced its new data governance solution, Amazon DataZone. Although the tool is currently in preview, it is now available in the US East (Northern Virginia), US West (Oregon) and Europe (Ireland) regions.

Data Governance is one of the hottest topics on the market today. Several companies around the international market have pointed problems in this area, such as:

  • Efficient data cataloging;
  • Discovery and documentation of data that gives users autonomy for decision making;
  • Data Literacy — sharing data knowledge across the organization allowing the operation to become increasingly data driven;
  • Data Quality — Correct, reliable and consistent data;
  • Data Availability — Data available at all times and failsafe processing and serving pipelines.

Then, a pool of tools appeared on the market with features that allow covering some of the challenges cited, especially those related to data cataloging. Informatica's tool is perhaps the best known among the licensed. Among the open source tools, I highlight Data Hub (www.datahubproject.io) developed on LinkedIn, Open Metadata (https://open-metadata.org/) and Amundsen (https://www.amundsen.io /) powered by Lyft. In addition to cataloging and discovering data artifacts, these tools allow for a view of data lineage, including technical documentation and business terms, and building relationships between data artifacts. Also, it is possible to register data owners, the people responsible for the data in those tools. This greatly facilitates access request and evaluation process (which today is a major bottleneck).

Amazon DataZone

I always tell my friends: "It was about time for AWS to launch its data governance tool!". DataZone comes with the main features needed for a good data catalog, in addition to security features and access management integrated into the AWS environment. Let's take a look.

The first step to use DataZone is the creation of a data domain (oriented to a business vertical, for example, sales, logistics, finance). It is curious to note that AWS took great care in showing all the access policies granted to the DataZone, as the tool will have great power in the environment.

View of the necessary accesses to the DataZone. Author's elaboration.

Policy details are available by clicking View permission details.

Details of DataZone access policies. Author's elaboration.

At this point, you can leverage IAM Identity Center (formerly AWS SSO) for granting domain access permissions. However, the domain must be deployed in the same region as the Identity Center. In my case, as you can see, this was not possible.

Access management configuration. Author's elaboration.

After a few seconds to create the domain, the access link becomes available. The tool's interface impresses with a beautiful design. The next step is to create a project within the existing domain. The tool already suggests a default profile that has a native connection with Athena, AWS Glue and S3.

DataZone project creation. Author's elaboration.

Once the project is ready, we will publish data for consultation in the tool. When the project was created, DataZone automatically created two databases in Amazon Athena, one with the _pub_db suffix and one with the _sub_db suffix. The "pub" database will be used for data producing teams to publish their tables. In my case, I already had some glue crawlers configured to automatically map tables in S3 - the 3 tables from Brazil's Higher Education Census (Censo da Educação Superior) 2019 and one table from Brazil's National Student Performance Exam 2017, all public datasets from the Ministry of Education. I just edited the crawlers so that these tables were available in the "pub" database. After that, it is necessary to publish the data in DataZone. In the project interface, we click on Publish Data and we see the screen below:

Data publication. Author's elaboration.

After filling in the settings and the job has been successfully executed, on the project page we can access the tables brought in the "publishing" menu.

Menu of published tables. Author's elaboration.

Note that all tables are still marked as draft, that is, they are not yet active to be viewed in the catalog. By clicking on the table name, it is possible to view various general information such as the location in S3 where the data is stored, region, data format, table schema and subscribers (if any). In the schema screen, we can edit the columns giving them a more readable name (the real name of the column is not changed and continues to be displayed in the interface) and a detailed description. A "readable" title and description can also be added to the table. After editing, it is necessary to click on set asset to active, so that the table can be consulted by other users of the catalog.

Table schema with readable column names and description. Author's elaboration.

After activating the tables, they are available for consultation in the Data catalog page and also via search.

Tables cataloged in the Sales domain. Author's elaboration.

To request access to these artifacts, other users registered in another project could open a registration request as consumers (subscribers). From there, data publishers can authorize requests and set granular per-table permissions for their consumption.

Final evaluation

DataZone is still in preview and is not intended by Amazon for production use. The tool already has several important features for data governance but there is still a lot to develop. Below, I write down some pros and cons of the tool so far.

Pros

  • Easy integration with the AWS environment;
  • Integrated security and access management with AWS Identity Center or IAM;
  • Easy metadata ingestion;
  • Creation of automated projects with CloudFormation (transparent to the user);
  • Efficient data search;
  • Glossary of business terms.

Cons

  • The tool still does not have data lineage implemented, a very important feature to understand the construction of KPIs;
  • It does not have integration with any data quality framework;
  • It is not possible to register other data artifacts such as dashboards, charts, pipelines, etc.

In fact, Amazon DataZone is still a tool under development and it has an enormous potential. I look forward to the following development steps of this promising tool.

Top comments (2)

Collapse
 
garryhammack profile image
Danny Acton

Data governance involves the management and control of an organization's data assets, ensuring their quality, security, compliance, and effective use across the organization and use papersessays.com/ site for quality work. Amazon DataZone is a service offered by AWS that helps organizations establish and enforce data governance policies, monitor data usage, and maintain data quality.

Collapse
 
dendihandian profile image
Dendi Handian

Seems like this is similar to Data Catalog in DataHub, which the main feature to document the data per business-unit or even across organization.