Medallion Architecture on BigQuery with dbt Core and Cloud Run

Not every data platform runs on a single cloud. In practice, I keep seeing organizations with CI/CD in Azure DevOps, compute on GCP, and production workloads scattered across providers. The architecture needs to work with this reality, not pretend it doesn’t exist.

Here’s a reference architecture I’ve designed and deployed for implementing the Medallion pattern on Google BigQuery — with dbt Core as the transformation engine, deployed via Cloud Run containers, and governed by Dataplex. It’s pragmatic, cross-cloud, and it works.

The Stack

• Transformation: dbt Core on Cloud Run
• Storage: BigQuery (all three layers)
• Orchestration: Cloud Scheduler
• CI/CD: Azure Repos + Azure Pipelines
• Governance: Dataplex

Yes, Azure Repos with GCP compute. Cross-cloud is messy. It’s also reality for many enterprises.

Data Sources

The typical enterprise feeds:

• ERP / SAP
• CRM
• APIs / SaaS platforms
• Files (landing in GCS buckets)

Ingestion uses Cloud Dataflow or Cloud Functions to land raw data into the Bronze layer.

The Three Layers

Bronze — Raw / Landing

• BigQuery raw_* datasets
• GCS Bucket (landing zone for files)
• Schema Registry for validation
• Data arrives as-is, no transformations

The rule: never modify Bronze data. It’s your audit trail.

Silver — Cleansed / Conformed

• BigQuery stg_* (staging) and int_* (intermediate) models
• dbt Core transformations: dedup, type casting, joins, null handling
• Data quality tests (dbt tests validate assumptions at every run)

The rule: Silver data has consistent types, no duplicates, and passes quality checks.

Gold — Business-Ready

• BigQuery mart_*, dim_*, fct_* models
• Materialized views and aggregates
• Semantic layer (metrics, KPIs, business definitions)

The rule: Gold models are what business users query directly. They should be named in business language, not technical jargon.

Compute: dbt Core on Cloud Run

dbt Core runs as a containerized job on Cloud Run:

• Triggered by Cloud Scheduler — runs on a cron schedule
• Processes Silver and Gold layers in dependency order
• Stateless — each run is fresh, no leftover state between executions
• Cost-efficient — you pay only for execution time, not for an always-on dbt server

The container image is built and deployed via Azure Pipelines.

CI/CD: Azure Repos + Pipelines

Source control lives in Azure Repos (Git). The pipeline:

1. Lint — dbt compile + sqlfluff
2. Test — dbt test against a development dataset
3. Deploy — Build container image, push to Artifact Registry, deploy to Cloud Run

This works because dbt Core is just Python + SQL. It doesn’t care where its Git repo lives.

Governance: Dataplex

Dataplex spans all three layers and provides:

• Data Catalog — metadata management, discoverability
• Data Quality — automated quality rules and monitoring
• Lineage — end-to-end tracking from source to Gold
• Policy Tags — column-level security classification
• Access Control — IAM-based permissions per dataset/table

Without governance, a data platform becomes a data swamp within 6 months.

Architecture Diagram

Sources → Ingestion → [Bronze] → [Silver] → [Gold] → Consumers
               ↑                      ↑           ↑
               |          ┌───────────┴───────────┘
               |          │
          Azure Repos → dbt Core (Cloud Run)
               CI/CD      (Cloud Scheduler)

         ╔══════════════════════════════════════════╗
         ║  Dataplex Governance (spans all layers)  ║
         ║  Catalog · Quality · Lineage · Policy    ║
         ╚══════════════════════════════════════════╝

Consumers

The Gold layer feeds:

• Looker / BI dashboards — self-service analytics
• ML / Vertex AI — feature store, training datasets
• Applications — APIs, microservices reading business-ready data
• Reverse ETL — syncing insights back to operational systems (CRM, ERP)

Key Takeaways

• The Medallion pattern (Bronze → Silver → Gold) is a separation of concerns for data — raw, cleaned, business-ready
• dbt Core on Cloud Run gives you serverless transformation without the cost of an always-on orchestrator
• Cross-cloud is real — Azure DevOps CI/CD with GCP compute works fine
• Dataplex (or equivalent) is not optional — without governance, data platforms decay fast
• Name your Gold models in business language — if analysts can’t find it, it doesn’t exist
• Never modify Bronze — it’s your immutable audit trail

Any opinions in this article are my own.