了解数据形式术语和身份验证流程 |通过卡比尔·阿坎德 | 2024 年 5 月 |走向数据科学

了解 Dataform 术语和身份验证流程：Dataform 是一项 GCP 服务，通过结合工程最佳实践和自动化工作流程编排来简化数据管道。通过在工作区中组织代码、管理配置和定义环境，Dataform 可以实现高效的数据转换和结构化数据输出。本文深入探讨了关键的 Dataform 术语，包括开发工作区、发布配置和工作流配置，并探讨了使用服务帐户和模拟的身份验证过程。

MLOps: Data Pipeline Orchestration

MLOps：数据管道编排

A Comprehensive Guide to Dataform Terminologies and Authentication Flow

数据形式术语和身份验证流程综合指南

Part 1 of Dataform 101: Fundamentals of a Single Repo, Multi-Environment Dataform with Least-Privilege Access Control and Infrastructure as Code Setup

Dataform 101 第 1 部分：单个存储库、具有最低权限访问控制和基础设施即代码设置的多环境数据表的基础知识

Introduction

介绍

Dataform is an innovative service offered by Google Cloud Platform (GCP) that empowers teams to develop and operationalize sophisticated, SQL-based data pipelines. This groundbreaking platform seamlessly integrates software engineering best practices into the data pipeline lifecycle, unlocking a world of possibilities, including testing, environments, version control, dependency management, orchestration, and automated documentation. Essentially, Dataform serves as a serverless, SQL workflow orchestration engine within the GCP ecosystem. It ingests raw data, transforms it with the utmost engineering rigor, and produces meticulously structured data ready for consumption.

Dataform 是 Google Cloud Platform (GCP) 提供的一项创新服务，使团队能够开发和运营复杂的、基于 SQL 的数据管道。这个突破性的平台将软件工程最佳实践无缝集成到数据管道生命周期中，开启了一个充满可能性的世界，包括测试、环境、版本控制、依赖管理、编排和自动化文档。本质上，Dataform 充当 GCP 生态系统中的无服务器 SQL 工作流编排引擎。它摄取原始数据，以最严格的工程严谨性对其进行转换，并生成可供使用的精心结构化的数据。

Dataform Terminologies

数据形式术语

Navigating the world of Dataform can be a daunting task, especially for those new to its terminologies. To bridge this knowledge gap, let's delve into some key terms that will serve as a compass on your Dataform journey:

探索 Dataform 的世界可能是一项艰巨的任务，尤其是对于那些不熟悉其术语的人来说。为了弥补这一知识差距，让我们深入研究一些关键术语，这些术语将作为您 Dataform 之旅的指南针：

Development Workspaces:

开发工作区：

Similar to local GitHub branches, Dataform development workspaces provide an editable copy of the main Dataform repository code. Within these workspaces, code development and experimentation flourish. When the time comes to share your masterpiece with the world, simply commit and push your changes, and voilà, a remote branch will emerge, mirroring your development workspace.

与本地 GitHub 分支类似，Dataform 开发工作区提供主 Dataform 存储库代码的可编辑副本。在这些工作空间中，代码开发和实验蓬勃发展。当需要与世界分享您的杰作时，只需提交并推送您的更改，瞧，就会出现一个远程分支，镜像您的开发工作区。

Release Configuration:

发布配置：

Dataform seamlessly blends .sqlx scripts with JavaScript (.js) for data transformations and logic. To ensure the codebase is standardized and reproducible, Dataform employs a compilation process that transforms it into a machine-readable JSON config file. The orchestrator behind this compilation process is none other than release configuration. At a predetermined time, Dataform will diligently check out the code from a remote repository (which can be configured to target any branch) and weave its transformation magic, generating the JSON config file.

Dataform 将 .sqlx 脚本与 JavaScript (.js) 无缝混合，以实现数据转换和逻辑。为了确保代码库标准化且可重复，Dataform 采用编译过程将其转换为机器可读的 JSON 配置文件。这个编译过程背后的协调器正是发布配置。在预定的时间，Dataform 会认真地从远程存储库（可以配置为针对任何分支）检查代码，并编织其转换魔法，生成 JSON 配置文件。

Workflow Configuration:

工作流程配置：

The output of release configuration – the JSON config file – serves as the foundation for workflow configuration. This configuration dictates when the config file should be executed, who should execute it, and where the config file's output should reside.

发布配置的输出（JSON 配置文件）作为工作流配置的基础。此配置指示何时应执行配置文件、谁应执行它以及配置文件的输出应驻留在何处。

Since workflow configuration relies heavily on the JSON config file generated by release configuration, it's imperative to ensure that it runs after release configuration. This ensures that the latest changes are reflected in the BigQuery (BQ) tables without delay. If scheduled simultaneously, workflow configuration may utilize the previous compilation, resulting in a lag in reflecting the most up-to-date changes in the BQ tables until the next workflow configuration run.

由于工作流配置严重依赖于发布配置生成的 JSON 配置文件，因此必须确保它在发布配置后运行。这可确保最新更改立即反映在 BigQuery (BQ) 表中。如果同时安排，工作流配置可能会利用先前的编译，从而导致在下一次工作流配置运行之前反映 BQ 表中的最新更改存在滞后。

Environments:

环境：

One of Dataform's most sought-after features is its ability to manifest code into multiple environments, such as development, staging, and production. This flexibility poses a question: should repositories be created in multiple environments or consolidated into a single environment? Google, in its infinite wisdom, has weighed the pros and cons in its Dataform best practices section. This article will demonstrate how to configure Dataform for staging and production environments, with data materialized into both environments from a single repository.

Dataform 最受欢迎的功能之一是它能够将代码体现到多种环境中，例如开发、登台和生产。这种灵活性提出了一个问题：存储库应该在多个环境中创建还是合并到单个环境中？谷歌以其无限的智慧，在其数据形式最佳实践部分权衡了利弊。本文将演示如何为临时环境和生产环境配置 Dataform，并将数据从单个存储库具体化到这两个环境中。

Authentication

验证

Authentication within Dataform can be a labyrinthine maze, especially when navigating multiple environments. To shed light on this intricate matter, we will delve into the specifics of staging and production environments.

Dataform 中的身份验证可能是一个迷宫，尤其是在多个环境中导航时。为了阐明这个复杂的问题，我们将深入研究暂存和生产环境的细节。

Dataform Authentication Flow:

数据形式验证流程：

The diagram below captures the intricate dance of authentication within Dataform:

下图展示了 Dataform 中复杂的身份验证过程：

[Image of Dataform authentication flow]

[Dataform 身份验证流程图]

Machine User:

机器用户：

Dataform requires credentials to access GitHub when retrieving code stored on a remote repository. While using individual credentials is an option, the recommended path is to employ a machine user within an organization. This strategy ensures that Dataform's pipeline orchestration remains independent of individual identities and is not susceptible to their departure or whims. Setting up a machine user involves creating an identity that is not tied to an individual and using it to establish a GitHub account. For Dataform, a personal access token (PAT) is generated for the machine user account and safely tucked away as a secret in GCP's secret manager. The machine user should also be added as an outside collaborator to the Dataform remote repository with read and write access. We will unravel how Dataform is configured to access this secret in the Terraform code. If the user opts to use their own identity instead of a machine user, a token should be generated using the steps outlined here.

当检索存储在远程存储库上的代码时，Dataform 需要凭据才能访问 GitHub。虽然可以选择使用个人凭据，但建议的途径是在组织内雇用计算机用户。此策略确保 Dataform 的管道编排保持独立于个人身份，并且不易受到个人身份的离开或突发奇想的影响。设置机器用户涉及创建一个与个人无关的身份并使用它来建立 GitHub 帐户。对于 Dataform，会为计算机用户帐户生成个人访问令牌 (PAT)，并将其作为秘密安全地隐藏在 GCP 的秘密管理器中。计算机用户还应作为外部协作者添加到具有读写访问权限的 Dataform 远程存储库。我们将揭示如何配置 Dataform 来访问 Terraform 代码中的此秘密。如果用户选择使用自己的身份而不是计算机用户，则应使用此处概述的步骤生成令牌。

GitHub Authentication Flow:

GitHub 身份验证流程：

Dataform leverages its default service account for implementation. When a Dataform action is initiated, it commences with the default service account. Assuming you have created a machine user, added them as a collaborator to the remote repository, and securely stored their PAT as a secret in GCP's secret manager, let's delve into the authentication process:

Dataform 利用其默认服务帐户进行实施。当启动 Dataform 操作时，它会从默认服务帐户开始。假设您已经创建了一个计算机用户，将他们作为协作者添加到远程存储库，并将其 PAT 作为秘密安全地存储在 GCP 的秘密管理器中，让我们深入研究身份验证过程：

1. Extracting the Secret: The default service account requires the secretAccessor role to extract the secret from the secret manager.
2. Impersonating the Machine User: Armed with the secret, the default service account can now impersonate the machine user. Since the machine user is a collaborator on the remote Git repository, the default service account inherits access to the repository.

[Image of GitHub authentication workflow]

提取机密：默认服务帐户需要 SecretAccessor 角色才能从机密管理器中提取机密。 模拟计算机用户：有了机密，默认服务帐户现在可以模拟计算机用户。由于机器用户是远程 Git 存储库上的协作者，因此默认服务帐户继承对存储库的访问权限。[GitHub 身份验证工作流程的图片]

Development Workspace Authentication:

开发工作区身份验证：

When execution is initiated from the development workspace, the default service account assumes the identity of the staging environment's custom service account to manifest the output within the staging environment. To successfully impersonate the staging environment's custom service account, the default service account must possess the iam.serviceAccountTokenCreator role. This role empowers the default service account to craft a short-lived token, akin to the PAT used to impersonate the machine user, for the staging custom service account and, thus, impersonate it. Consequently, the staging custom service account inherits all the necessary permissions to modify BQ tables, which are then inherited by the default service account when it impersonates the staging custom service account.

从开发工作区启动执行时，默认服务帐户采用暂存环境的自定义服务帐户的身份，以在暂存环境中显示输出。要成功模拟暂存环境的自定义服务帐户，默认服务帐户必须拥有 iam.serviceAccountTokenCreator 角色。此角色使默认服务帐户能够为暂存自定义服务帐户制作一个短期令牌（类似于用于模拟计算机用户的 PAT），从而模拟它。因此，暂存自定义服务帐户继承修改 BQ 表所需的所有权限，然后默认服务帐户在模拟暂存自定义服务帐户时继承这些权限。

Workflow Configuration Authentication:

工作流程配置身份验证：

After successfully retrieving the code from the repository, release configuration generates a compiled config JSON file, which workflow configurations utilize to generate data. To commit this data to production BQ tables, the default service account requires the iam.serviceAccountTokenCreator role on the production custom service account. Similar to the process employed for the staging custom service account, the production service account is granted all the necessary permissions to modify production environment BQ tables, which are inherited by the default service account when it impersonates the production custom service account.

从存储库成功检索代码后，发布配置会生成一个已编译的配置 JSON 文件，工作流配置利用该文件来生成数据。要将这些数据提交到生产 BQ 表，默认服务帐户需要生产自定义服务帐户上的 iam.serviceAccountTokenCreator 角色。与暂存自定义服务帐户采用的过程类似，生产服务帐户被授予修改生产环境 BQ 表的所有必要权限，这些权限由默认服务帐户在模拟生产自定义服务帐户时继承。

Conclusion

结论

In summary, the default service account plays a pivotal role in Dataform's authentication architecture. It assumes the identity of the machine user to authenticate with GitHub as a collaborator, leveraging the machine user's PAT. It also authenticates with the staging and production environments by impersonating their respective custom service accounts using short-lived tokens generated with the serviceAccountTokenCreator role. Armed with this newfound understanding, we can confidently embark on the journey of provisioning Dataform within GCP using Terraform. Stay tuned for Part 2 of this article, where we will tackle this very subject. In the meantime, feel free to explore the code repository for further insights.

总之，默认服务帐户在 Dataform 的身份验证架构中起着关键作用。它假定机器用户的身份，利用机器用户的 PAT 作为协作者向 GitHub 进行身份验证。它还通过使用 serviceAccountTokenCreator 角色生成的短期令牌来模拟各自的自定义服务帐户，从而对登台和生产环境进行身份验证。有了这种新的理解，我们就可以自信地踏上使用 Terraform 在 GCP 内配置 Dataform 的旅程。请继续关注本文的第 2 部分，我们将在其中讨论这个主题。同时，请随意探索代码存储库以获取更多见解。

References:

参考：

• [Dataform: Google Cloud](https://cloud.google.com/dataform?hl=en)
• [Dataform Migration: Google Cloud](https://cloud.google.com/dataform/docs/migration)
• [Dataform Best Practices: Google Cloud](https://cloud.google.com/dataform/docs/best-practices)

[数据形式：Google Cloud](https://cloud.google.com/dataform?hl=en)[数据形式迁移：Google Cloud](https://cloud.google.com/dataform/docs/migration)[Dataform 最佳实践：Google Cloud](https://cloud.google.com/dataform/docs/best-practices)