Pinterest – Introducing dbt Company-Wide via Platform Partnership
S: 2023 | Staff Eng. Infra Gov. Pinterest had zero dbt adoption company-wide. 150+ tables, monolithic templated SQL (30+ CTEs, 500+ lines), no unit testing. System allocating $700M/year in cloud spend couldn’t be safely changed. C-level reports had been wrong.
T: Introduce modern data modeling discipline to Pinterest and make it viable within Pinterest’s proprietary orchestration environment (Spinner, a heavily customized Airflow fork with capacity management and tiered scheduling).
A1 – Diagnose: Identified missing modeling discipline as root cause (not tactical bugs). Wrote influential paper proposing a new logical model that decoupled attribution from aggregation—so the system could survive re-orgs without rework.
A2 – Pilot: Brought dbt to the team; proved value on first platform migration. Demonstrated testability, modularity, and explicit dependency management replacing monolithic SQL.
A3 – Partnership: Engaged Data Platform team; they built initial DAG renderer (parses dbt manifest, generates Spinner DAGs). I extended the renderer with Iceberg WAP support and additional metadata for our use cases. Could not use off-the-shelf tools like Cosmos due to Spinner’s custom scheduling.
A4 – Evangelize: Shared results across Infra Governance; other teams adopted the dbt-on-Spinner pattern for their own data pipelines.
R: dbt became standard for Infra Governance data pipelines. 6 platforms migrated in 6 months with 2 engineers. Pattern reusable beyond my team. PagerDuty pages dropped from 200/month to 5/month. SLA breaches went to zero.
TRAITS: Strategy • Influence • Evangelism (Q 3 12 38 40)