Data Engineer (GCP) Roadmap

What a GCP Data Engineer Actually Does beginner: The real day to day of the role: building pipelines, fixing broken ones, and where it differs from analytics or ML engineering.

• ETL vs ELT: Whether transformation happens before or after loading into the warehouse.
• Batch vs Streaming: Processing data in scheduled chunks versus continuously as it arrives.
• Data Engineer vs Analytics Engineer: Pipeline and infra ownership versus modeling data already in the warehouse.
• OLTP vs OLAP: Transactional systems built for writes versus analytical systems built for reads.

GCP Core Services Landscape beginner: How GCP's projects, regions and IAM hierarchy fit together before touching any individual data service.

• Projects & Billing: The container that scopes resources, permissions and cost in GCP.
• Regions & Zones: Physical locations where GCP resources actually run.
• IAM Basics: Who can do what, controlled through roles bound to identities.
• gcloud CLI & Console: The two main ways engineers interact with GCP resources.

Advanced SQL for Analytics Workloads beginner: Window functions, CTEs and query patterns that show up constantly in BigQuery transformation logic.

• Window Functions: Calculations across rows related to the current row, like running totals.
• Common Table Expressions: Named temporary result sets that make complex queries readable.
• Query Execution Plans: How the database decides to run your query and where time is spent.

Python for Data Pipelines beginner: The specific Python patterns pipelines actually use: typed functions, generators and working with structured records.

• Generators & Iterators: Processing data lazily instead of loading everything into memory.
• Type Hints: Annotating expected types, which pipeline frameworks rely on heavily.
• Working with JSON/Avro: The structured formats most pipeline data actually arrives in.

Data Modeling Fundamentals beginner: How to shape tables before you build pipelines, since a bad schema breaks every downstream step.

• Star Schema: A fact table surrounded by dimension tables, the standard analytics layout.
• Normalization vs Denormalization: Trading storage efficiency for query simplicity.
• Slowly Changing Dimensions: How to track a dimension's values as they change over time.

BigQuery Fundamentals beginner: How BigQuery actually executes queries under the hood, which explains most of its cost and performance behavior.

• Columnar Storage: Why BigQuery only reads the columns your query actually references.
• Slots & On-Demand Pricing: The compute unit BigQuery bills against, and the two ways to pay for it.
• Datasets & Tables: How data is organized and permissioned within a BigQuery project.

Partitioning, Clustering & Query Optimization intermediate: The techniques that turn an expensive full table scan into a query that costs cents.

• Table Partitioning: Splitting a table by date or integer range so queries scan less data.
• Clustering: Sorting data within partitions to speed up filtered queries.
• Materialized Views: Precomputed query results BigQuery keeps automatically up to date.
• BI Engine: In-memory acceleration layer for fast dashboard queries.

BigQuery Advanced Features intermediate: The features that go beyond plain SQL: ML inside the warehouse, external tables, and semi-structured data.

• BigQuery ML: Training and running ML models directly with SQL syntax.
• External & Federated Tables: Querying data in Cloud Storage or other sources without loading it in.
• Nested & Repeated Fields: Storing semi-structured, JSON-like data natively in a BigQuery table.
• BigQuery Storage API: High throughput programmatic read access to BigQuery data.

Cloud Storage & Data Lake Design beginner: Where raw data lands before it's structured, and how to organize a bucket so it doesn't become a dumping ground.

• Storage Classes: Standard, Nearline, Coldline and Archive, priced by access frequency.
• Bucket & Object Lifecycle Rules: Automatically moving or deleting data based on age or conditions.
• Data Lake vs Lakehouse: Raw file storage versus a lake with warehouse-like structure on top.

Apache Beam Programming Model intermediate: The unified batch and streaming abstraction that Dataflow is built on top of.

• PCollections: Beam's distributed, immutable dataset abstraction.
• Transforms & Pipelines: The operations applied to data and how they chain into a pipeline.
• Windowing: Grouping unbounded data into finite chunks for processing.
• Triggers: Rules that decide when a window's results are emitted.

Building Batch Pipelines on Dataflow intermediate: Taking a Beam pipeline from local development to a running, autoscaled Dataflow job.

• Dataflow Runner: The GCP service that executes Beam pipelines at scale.
• Autoscaling Workers: How Dataflow adds and removes compute based on backlog.
• Dataflow Templates: Reusable, parameterized pipeline definitions for common jobs.

Debugging & Optimizing Dataflow Jobs advanced: Reading the Dataflow job graph to find the actual bottleneck instead of guessing.

• Job Graph & Stages: The visual execution plan Dataflow shows for a running job.
• Data Skew: Uneven distribution of work across workers that slows a whole job down.
• Side Inputs: Passing a small auxiliary dataset into every element of a main pipeline.

Pub/Sub Messaging Fundamentals intermediate: How GCP's messaging backbone decouples data producers from the pipelines that consume them.

• Topics & Subscriptions: Where publishers send messages and where consumers read them from.
• At-Least-Once Delivery: Pub/Sub's delivery guarantee and why consumers must handle duplicates.
• Push vs Pull Subscriptions: Whether Pub/Sub sends messages to you or you fetch them.

Streaming Pipelines with Dataflow advanced: Connecting Pub/Sub to Dataflow to process events within seconds of them arriving.

• Unbounded PCollections: Beam's representation of a never-ending stream of data.
• Watermarks: Dataflow's estimate of how complete a window's data is at a point in time.
• Late Data Handling: Deciding what happens to events that arrive after their window closed.
• Exactly-Once Processing: Ensuring each event affects the output exactly one time despite retries.

Streaming Data into BigQuery intermediate: The specific patterns for landing real time events into queryable BigQuery tables.

• Streaming Inserts API: Writing rows to BigQuery in near real time as they arrive.
• Storage Write API: The newer, higher throughput and lower cost streaming ingestion path.
• Deduplication Strategies: Handling duplicate events from at-least-once delivery.

Apache Spark Fundamentals intermediate: The distributed processing engine you reach for when SQL alone stops being enough.

• RDDs & DataFrames: Spark's core distributed data abstractions.
• Lazy Evaluation: Spark builds a plan and only executes it when a result is requested.
• PySpark Basics: Writing Spark jobs using Python instead of Scala.

Running Spark on Dataproc intermediate: GCP's managed Hadoop and Spark service, and when to reach for it over BigQuery.

• Ephemeral Clusters: Spinning up a Dataproc cluster only for the duration of a job.
• Dataproc Serverless: Running Spark jobs without managing cluster infrastructure at all.
• Dataproc vs BigQuery vs Dataflow: When to pick each of GCP's three main processing engines.

dbt Core Fundamentals on BigQuery intermediate: Version controlled, testable SQL transformations that turn raw BigQuery tables into trusted models.

• Models & Materializations: dbt's SQL files and how they get turned into tables or views.
• Sources & Refs: How dbt tracks dependencies between raw data and transformed models.
• Jinja Templating: dbt's templating language for writing dynamic, reusable SQL.

dbt Testing & Documentation intermediate: Catching broken data before it reaches a dashboard, and making models self-documenting.

• Generic & Singular Tests: Built-in and custom assertions dbt runs against your data.
• dbt Docs & Lineage Graph: Auto-generated documentation showing how models connect.
• Source Freshness Checks: Alerting when upstream source data stops updating as expected.

Apache Airflow DAG Fundamentals intermediate: How to express a pipeline as a graph of dependent tasks instead of a linear script.

• DAGs & Operators: The graph structure and the building blocks that make up each task.
• Task Dependencies: Defining which tasks must finish before others can start.
• Scheduling & Backfills: Running a DAG on a schedule, and re-running it for past dates.

Cloud Composer Environment Management intermediate: Running managed Airflow on GCP, including the parts that differ from self-hosted Airflow.

• Composer Environments: The managed Airflow instance GCP provisions and maintains.
• GCP Operators: Pre-built Airflow tasks for triggering BigQuery, Dataflow and other GCP services.
• Environment Variables & Connections: Securely passing config and credentials into DAGs.

Pipeline Monitoring & Alerting advanced: Knowing a pipeline broke before an analyst tells you the dashboard looks wrong.

• Cloud Monitoring & Logging: GCP's native tools for tracking pipeline health and errors.
• SLA Misses & Alerts: Getting notified when a DAG doesn't finish within its expected time.
• Data Quality Monitoring: Detecting anomalies in the data itself, not just pipeline failures.

IAM & Data Governance on GCP intermediate: Controlling who can see and change data, which matters the moment more than one team touches a project.

• Least Privilege Roles: Granting only the exact permissions a user or service needs.
• Service Accounts: Non-human identities that pipelines use to access GCP resources.
• Column & Row Level Security: Restricting access to specific parts of a BigQuery table.

Data Catalog & Lineage advanced: Making data discoverable and traceable across an organization instead of tribal knowledge.

• Dataplex: GCP's unified data governance and cataloging service.
• Metadata Tagging: Labeling datasets with ownership, sensitivity and business context.
• PII Detection with DLP: Automatically finding sensitive data using Cloud Data Loss Prevention.

Terraform for GCP Infrastructure advanced: Defining BigQuery datasets, Composer environments and IAM bindings as versioned code instead of console clicks.

• Providers & Resources: How Terraform declares and manages GCP infrastructure.
• State Management: How Terraform tracks what infrastructure currently exists.
• Modules for Data Infra: Reusable Terraform configurations for common data platform pieces.

CI/CD for Data Pipelines with Cloud Build advanced: Automatically testing and deploying dbt models, Dataflow jobs and DAGs on every code change.

• Build Triggers: Automatically starting a pipeline deployment on a git push.
• Testing dbt & DAGs in CI: Running automated checks before changes reach production.
• Artifact Registry: Storing container images and packages used by pipeline deployments.