了解資料形式術語和身份驗證流程 |透過卡比爾·阿坎德 | 2024 年 5 月 |走向資料科學

了解 Dataform 術語和身分驗證流程：Dataform 是一項 GCP 服務，透過結合工程最佳實務和自動化工作流程編排來簡化資料管道。透過在工作區中組織程式碼、管理配置和定義環境，Dataform 可以實現高效的資料轉換和結構化資料輸出。本文深入探討了關鍵的 Dataform 術語，包括開發工作區、發布配置和工作流程配置，並探討了使用服務帳戶和模擬的身份驗證流程。

MLOps: Data Pipeline Orchestration

MLOps：資料管道編排

A Comprehensive Guide to Dataform Terminologies and Authentication Flow

資料形式術語和身分驗證流程綜合指南

Part 1 of Dataform 101: Fundamentals of a Single Repo, Multi-Environment Dataform with Least-Privilege Access Control and Infrastructure as Code Setup

Dataform 101 第 1 部分：單一儲存庫、具有最低權限存取控制和基礎架構即程式碼設定的多環境資料表的基礎知識

Introduction

介紹

Dataform is an innovative service offered by Google Cloud Platform (GCP) that empowers teams to develop and operationalize sophisticated, SQL-based data pipelines. This groundbreaking platform seamlessly integrates software engineering best practices into the data pipeline lifecycle, unlocking a world of possibilities, including testing, environments, version control, dependency management, orchestration, and automated documentation. Essentially, Dataform serves as a serverless, SQL workflow orchestration engine within the GCP ecosystem. It ingests raw data, transforms it with the utmost engineering rigor, and produces meticulously structured data ready for consumption.

Dataform 是 Google Cloud Platform (GCP) 提供的創新服務，使團隊能夠開發和營運複雜的、基於 SQL 的資料管道。這個突破性的平台將軟體工程最佳實踐無縫整合到資料管道生命週期中，開啟了一個充滿可能性的世界，包括測試、環境、版本控制、依賴管理、編排和自動化文件。本質上，Dataform 充當 GCP 生態系統中的無伺服器 SQL 工作流程編排引擎。它攝取原始數據，以最嚴格的工程嚴謹性對其進行轉換，並產生可供使用的精心結構化的數據。

Dataform Terminologies

資料形式術語

Navigating the world of Dataform can be a daunting task, especially for those new to its terminologies. To bridge this knowledge gap, let's delve into some key terms that will serve as a compass on your Dataform journey:

探索 Dataform 的世界可能是一項艱鉅的任務，尤其是對於不熟悉其術語的人來說。為了彌補這一知識差距，讓我們深入研究一些關鍵術語，這些術語將作為您 Dataform 之旅的指南針：

Development Workspaces:

開發工作區：

Similar to local GitHub branches, Dataform development workspaces provide an editable copy of the main Dataform repository code. Within these workspaces, code development and experimentation flourish. When the time comes to share your masterpiece with the world, simply commit and push your changes, and voilà, a remote branch will emerge, mirroring your development workspace.

與本機 GitHub 分支類似，Dataform 開發工作區提供主 Dataform 儲存庫程式碼的可編輯副本。在這些工作空間中，程式碼開發和實驗蓬勃發展。當需要與世界分享您的傑作時，只需提交並推送您的更改，瞧，就會出現一個遠端分支，鏡像您的開發工作區。

Release Configuration:

發布配置：

Dataform seamlessly blends .sqlx scripts with JavaScript (.js) for data transformations and logic. To ensure the codebase is standardized and reproducible, Dataform employs a compilation process that transforms it into a machine-readable JSON config file. The orchestrator behind this compilation process is none other than release configuration. At a predetermined time, Dataform will diligently check out the code from a remote repository (which can be configured to target any branch) and weave its transformation magic, generating the JSON config file.

Dataform 將 .sqlx 腳本與 JavaScript (.js) 無縫混合，以實現資料轉換和邏輯。為了確保程式碼庫標準化且可重複，Dataform 採用編譯過程將其轉換為機器可讀的 JSON 設定檔。這個編譯過程背後的協調器正是發布配置。在預定的時間，Dataform 會認真地從遠端儲存庫（可以配置為針對任何分支）檢查程式碼，並編織其轉換魔法，產生 JSON 設定檔。

Workflow Configuration:

工作流程配置：

The output of release configuration – the JSON config file – serves as the foundation for workflow configuration. This configuration dictates when the config file should be executed, who should execute it, and where the config file's output should reside.

發布配置的輸出（JSON 設定檔）作為工作流程配置的基礎。此配置指示何時應執行設定檔、誰應執行它以及設定檔的輸出應駐留在何處。

Since workflow configuration relies heavily on the JSON config file generated by release configuration, it's imperative to ensure that it runs after release configuration. This ensures that the latest changes are reflected in the BigQuery (BQ) tables without delay. If scheduled simultaneously, workflow configuration may utilize the previous compilation, resulting in a lag in reflecting the most up-to-date changes in the BQ tables until the next workflow configuration run.

由於工作流程配置嚴重依賴發布配置生成的 JSON 配置文件，因此必須確保它在發布配置後運行。這可確保最新變更立即反映在 BigQuery (BQ) 表中。如果同時安排，工作流程配置可能會利用先前的編譯，從而導致在下一次工作流程配置運行之前反映 BQ 表中的最新變更存在滯後。

Environments:

環境：

One of Dataform's most sought-after features is its ability to manifest code into multiple environments, such as development, staging, and production. This flexibility poses a question: should repositories be created in multiple environments or consolidated into a single environment? Google, in its infinite wisdom, has weighed the pros and cons in its Dataform best practices section. This article will demonstrate how to configure Dataform for staging and production environments, with data materialized into both environments from a single repository.

Dataform 最受歡迎的功能之一是它能夠將程式碼體現到多種環境中，例如開發、登台和生產。這種靈活性提出了一個問題：儲存庫應該在多個環境中建立還是合併到單一環境中？谷歌以其無限的智慧，在其數據形式最佳實踐部分權衡了利弊。本文將示範如何為臨時環境和生產環境配置 Dataform，並將資料從單一儲存庫具體化到這兩個環境中。

Authentication

驗證

Authentication within Dataform can be a labyrinthine maze, especially when navigating multiple environments. To shed light on this intricate matter, we will delve into the specifics of staging and production environments.

Dataform 中的身份驗證可能是一個迷宮，尤其是在多個環境中導航時。為了闡明這個複雜的問題，我們將深入研究暫存和生產環境的細節。

Dataform Authentication Flow:

資料形式驗證流程：

The diagram below captures the intricate dance of authentication within Dataform:

下圖展示了 Dataform 中複雜的身份驗證過程：

[Image of Dataform authentication flow]

[Dataform 驗證流程圖]

Machine User:

機器使用者：

Dataform requires credentials to access GitHub when retrieving code stored on a remote repository. While using individual credentials is an option, the recommended path is to employ a machine user within an organization. This strategy ensures that Dataform's pipeline orchestration remains independent of individual identities and is not susceptible to their departure or whims. Setting up a machine user involves creating an identity that is not tied to an individual and using it to establish a GitHub account. For Dataform, a personal access token (PAT) is generated for the machine user account and safely tucked away as a secret in GCP's secret manager. The machine user should also be added as an outside collaborator to the Dataform remote repository with read and write access. We will unravel how Dataform is configured to access this secret in the Terraform code. If the user opts to use their own identity instead of a machine user, a token should be generated using the steps outlined here.

當檢索儲存在遠端儲存庫上的程式碼時，Dataform 需要憑證才能存取 GitHub。雖然可以選擇使用個人憑證，但建議的途徑是在組織內僱用電腦使用者。此策略確保 Dataform 的管道編排保持獨立於個人身份，並且不易受到個人身份的離開或突發奇想的影響。設定機器使用者涉及建立一個與個人無關的身份並使用它來建立 GitHub 帳戶。對於 Dataform，會為電腦使用者帳戶產生個人存取權杖 (PAT)，並將其作為秘密安全地隱藏在 GCP 的秘密管理器中。電腦使用者也應作為外部協作者新增至具有讀寫存取權限的 Dataform 遠端儲存庫。我們將揭示如何配置 Dataform 來存取 Terraform 程式碼中的此秘密。如果用戶選擇使用自己的身分而不是電腦用戶，則應使用此處概述的步驟產生令牌。

GitHub Authentication Flow:

GitHub 身份驗證流程：

Dataform leverages its default service account for implementation. When a Dataform action is initiated, it commences with the default service account. Assuming you have created a machine user, added them as a collaborator to the remote repository, and securely stored their PAT as a secret in GCP's secret manager, let's delve into the authentication process:

Dataform 利用其預設服務帳戶進行實作。當啟動 Dataform 操作時，它會從預設服務帳戶開始。假設您已經創建了一個電腦用戶，將他們作為協作者添加到遠端儲存庫，並將其 PAT 作為秘密安全地儲存在 GCP 的秘密管理器中，讓我們深入研究身份驗證過程：

1. Extracting the Secret: The default service account requires the secretAccessor role to extract the secret from the secret manager.
2. Impersonating the Machine User: Armed with the secret, the default service account can now impersonate the machine user. Since the machine user is a collaborator on the remote Git repository, the default service account inherits access to the repository.

[Image of GitHub authentication workflow]

提取機密：預設服務帳戶需要 SecretAccessor 角色才能從機密管理器中提取機密。由於機器使用者是遠端 Git 儲存庫上的協作者，因此預設服務帳戶繼承對儲存庫的存取權。

Development Workspace Authentication:

開發工作區身份驗證：

When execution is initiated from the development workspace, the default service account assumes the identity of the staging environment's custom service account to manifest the output within the staging environment. To successfully impersonate the staging environment's custom service account, the default service account must possess the iam.serviceAccountTokenCreator role. This role empowers the default service account to craft a short-lived token, akin to the PAT used to impersonate the machine user, for the staging custom service account and, thus, impersonate it. Consequently, the staging custom service account inherits all the necessary permissions to modify BQ tables, which are then inherited by the default service account when it impersonates the staging custom service account.

從開發工作區啟動執行時，預設服務帳戶採用暫存環境的自訂服務帳戶的身份，以在暫存環境中顯示輸出。若要成功模擬暫存環境的自訂服務帳戶，預設服務帳戶必須擁有 iam.serviceAccountTokenCreator 角色。此角色使預設服務帳戶能夠為暫存自訂服務帳戶製作一個短期令牌（類似於用於模擬電腦使用者的 PAT），從而模擬它。因此，暫存自訂服務帳戶繼承修改 BQ 表所需的所有權限，然後預設服務帳戶在模擬暫存自訂服務帳戶時繼承這些權限。

Workflow Configuration Authentication:

工作流程配置身份驗證：

After successfully retrieving the code from the repository, release configuration generates a compiled config JSON file, which workflow configurations utilize to generate data. To commit this data to production BQ tables, the default service account requires the iam.serviceAccountTokenCreator role on the production custom service account. Similar to the process employed for the staging custom service account, the production service account is granted all the necessary permissions to modify production environment BQ tables, which are inherited by the default service account when it impersonates the production custom service account.

從儲存庫成功檢索程式碼後，發布配置會產生一個已編譯的配置 JSON 文件，工作流程配置利用該文件來產生資料。要將這些資料提交到生產 BQ 表，預設服務帳戶需要生產自訂服務帳戶上的 iam.serviceAccountTokenCreator 角色。與暫存自訂服務帳戶採用的過程類似，生產服務帳戶被授予修改生產環境 BQ 表的所有必要權限，這些權限由預設服務帳戶在模擬生產自訂服務帳戶時繼承。

Conclusion

結論

In summary, the default service account plays a pivotal role in Dataform's authentication architecture. It assumes the identity of the machine user to authenticate with GitHub as a collaborator, leveraging the machine user's PAT. It also authenticates with the staging and production environments by impersonating their respective custom service accounts using short-lived tokens generated with the serviceAccountTokenCreator role. Armed with this newfound understanding, we can confidently embark on the journey of provisioning Dataform within GCP using Terraform. Stay tuned for Part 2 of this article, where we will tackle this very subject. In the meantime, feel free to explore the code repository for further insights.

總之，預設服務帳戶在 Dataform 的身份驗證架構中扮演關鍵角色。它假定機器使用者的身份，利用機器使用者的 PAT 作為協作者向 GitHub 進行身份驗證。它還透過使用 serviceAccountTokenCreator 角色產生的短期令牌來模擬各自的自訂服務帳戶，從而對登台和生產環境進行身份驗證。有了這種新的理解，我們就可以自信地踏上使用 Terraform 在 GCP 內配置 Dataform 的旅程。請繼續關注本文的第 2 部分，我們將在其中討論這個主題。同時，請隨意探索程式碼儲存庫以獲取更多見解。

References:

參考：

• [Dataform: Google Cloud](https://cloud.google.com/dataform?hl=en)
• [Dataform Migration: Google Cloud](https://cloud.google.com/dataform/docs/migration)
• [Dataform Best Practices: Google Cloud](https://cloud.google.com/dataform/docs/best-practices)

[資料形式：Google Cloud](https://cloud.google.com/dataform?hl=en)[資料形式遷移：Google Cloud](https://cloud.google.com/dataform/docs/migration)[ Dataform 最佳實務：Google Cloud](https://cloud.google.com/dataform/docs/best-practices)