Understanding Dataform Terminologies And Authentication Flow | by Kabeer Akande | May, 2024 | Towards Data Science

Understanding Dataform Terminologies and Authentication Flow: Dataform, a GCP service, streamlines data pipelines by incorporating engineering best practices and automating workflow orchestration. By organizing code in workspaces, managing configurations, and defining environments, Dataform enables efficient data transformation and structured data output. This article delves into key Dataform terminologies, including development workspaces, release configurations, and workflow configurations, and explores the authentication process using service accounts and impersonation.

MLOps: Data Pipeline Orchestration

A Comprehensive Guide to Dataform Terminologies and Authentication Flow

Part 1 of Dataform 101: Fundamentals of a Single Repo, Multi-Environment Dataform with Least-Privilege Access Control and Infrastructure as Code Setup

Introduction

Dataform is an innovative service offered by Google Cloud Platform (GCP) that empowers teams to develop and operationalize sophisticated, SQL-based data pipelines. This groundbreaking platform seamlessly integrates software engineering best practices into the data pipeline lifecycle, unlocking a world of possibilities, including testing, environments, version control, dependency management, orchestration, and automated documentation. Essentially, Dataform serves as a serverless, SQL workflow orchestration engine within the GCP ecosystem. It ingests raw data, transforms it with the utmost engineering rigor, and produces meticulously structured data ready for consumption.

Dataform Terminologies

Navigating the world of Dataform can be a daunting task, especially for those new to its terminologies. To bridge this knowledge gap, let's delve into some key terms that will serve as a compass on your Dataform journey:

Development Workspaces:

Similar to local GitHub branches, Dataform development workspaces provide an editable copy of the main Dataform repository code. Within these workspaces, code development and experimentation flourish. When the time comes to share your masterpiece with the world, simply commit and push your changes, and voilà, a remote branch will emerge, mirroring your development workspace.

Release Configuration:

Dataform seamlessly blends .sqlx scripts with JavaScript (.js) for data transformations and logic. To ensure the codebase is standardized and reproducible, Dataform employs a compilation process that transforms it into a machine-readable JSON config file. The orchestrator behind this compilation process is none other than release configuration. At a predetermined time, Dataform will diligently check out the code from a remote repository (which can be configured to target any branch) and weave its transformation magic, generating the JSON config file.

Workflow Configuration:

The output of release configuration – the JSON config file – serves as the foundation for workflow configuration. This configuration dictates when the config file should be executed, who should execute it, and where the config file's output should reside.

Since workflow configuration relies heavily on the JSON config file generated by release configuration, it's imperative to ensure that it runs after release configuration. This ensures that the latest changes are reflected in the BigQuery (BQ) tables without delay. If scheduled simultaneously, workflow configuration may utilize the previous compilation, resulting in a lag in reflecting the most up-to-date changes in the BQ tables until the next workflow configuration run.

Environments:

One of Dataform's most sought-after features is its ability to manifest code into multiple environments, such as development, staging, and production. This flexibility poses a question: should repositories be created in multiple environments or consolidated into a single environment? Google, in its infinite wisdom, has weighed the pros and cons in its Dataform best practices section. This article will demonstrate how to configure Dataform for staging and production environments, with data materialized into both environments from a single repository.

Authentication

Authentication within Dataform can be a labyrinthine maze, especially when navigating multiple environments. To shed light on this intricate matter, we will delve into the specifics of staging and production environments.

Dataform Authentication Flow:

The diagram below captures the intricate dance of authentication within Dataform:

[Image of Dataform authentication flow]

Machine User:

Dataform requires credentials to access GitHub when retrieving code stored on a remote repository. While using individual credentials is an option, the recommended path is to employ a machine user within an organization. This strategy ensures that Dataform's pipeline orchestration remains independent of individual identities and is not susceptible to their departure or whims. Setting up a machine user involves creating an identity that is not tied to an individual and using it to establish a GitHub account. For Dataform, a personal access token (PAT) is generated for the machine user account and safely tucked away as a secret in GCP's secret manager. The machine user should also be added as an outside collaborator to the Dataform remote repository with read and write access. We will unravel how Dataform is configured to access this secret in the Terraform code. If the user opts to use their own identity instead of a machine user, a token should be generated using the steps outlined here.

GitHub Authentication Flow:

Dataform leverages its default service account for implementation. When a Dataform action is initiated, it commences with the default service account. Assuming you have created a machine user, added them as a collaborator to the remote repository, and securely stored their PAT as a secret in GCP's secret manager, let's delve into the authentication process:

1. Extracting the Secret: The default service account requires the secretAccessor role to extract the secret from the secret manager.
2. Impersonating the Machine User: Armed with the secret, the default service account can now impersonate the machine user. Since the machine user is a collaborator on the remote Git repository, the default service account inherits access to the repository.

[Image of GitHub authentication workflow]

Development Workspace Authentication:

When execution is initiated from the development workspace, the default service account assumes the identity of the staging environment's custom service account to manifest the output within the staging environment. To successfully impersonate the staging environment's custom service account, the default service account must possess the iam.serviceAccountTokenCreator role. This role empowers the default service account to craft a short-lived token, akin to the PAT used to impersonate the machine user, for the staging custom service account and, thus, impersonate it. Consequently, the staging custom service account inherits all the necessary permissions to modify BQ tables, which are then inherited by the default service account when it impersonates the staging custom service account.

Workflow Configuration Authentication:

After successfully retrieving the code from the repository, release configuration generates a compiled config JSON file, which workflow configurations utilize to generate data. To commit this data to production BQ tables, the default service account requires the iam.serviceAccountTokenCreator role on the production custom service account. Similar to the process employed for the staging custom service account, the production service account is granted all the necessary permissions to modify production environment BQ tables, which are inherited by the default service account when it impersonates the production custom service account.

Conclusion

In summary, the default service account plays a pivotal role in Dataform's authentication architecture. It assumes the identity of the machine user to authenticate with GitHub as a collaborator, leveraging the machine user's PAT. It also authenticates with the staging and production environments by impersonating their respective custom service accounts using short-lived tokens generated with the serviceAccountTokenCreator role. Armed with this newfound understanding, we can confidently embark on the journey of provisioning Dataform within GCP using Terraform. Stay tuned for Part 2 of this article, where we will tackle this very subject. In the meantime, feel free to explore the code repository for further insights.

References:

• [Dataform: Google Cloud](https://cloud.google.com/dataform?hl=en)
• [Dataform Migration: Google Cloud](https://cloud.google.com/dataform/docs/migration)
• [Dataform Best Practices: Google Cloud](https://cloud.google.com/dataform/docs/best-practices)