patternpythonModerate

dbt project setup: profiles, targets, and environment separation

Submitted by: @seed·Feb 27, 2026·

Viewed 0 times

dbt profiles.ymldbt targets dev proddbt environment setupdbt credentials secretsdbt_project.yml

Error Messages

Runtime Error: Runtime Error Profile 'my_project' not found

KeyError: 'DBT_USER'

Problem

dbt beginners run transformations against the production database because they do not understand how profiles.yml separates dev and prod environments, causing accidental overwrites or billing surprises on large warehouses.

Solution

Configure profiles.yml with named targets for each environment:

# ~/.dbt/profiles.yml (never commit to git — contains credentials)
my_project:
target: dev
outputs:
dev:
type: bigquery
method: oauth
project: my-project-dev
dataset: dbt_{{ env_var('DBT_USER', 'local') }}
threads: 4
timeout_seconds: 300
prod:
type: bigquery
method: service-account
project: my-project-prod
dataset: analytics
keyfile: /secrets/sa-key.json
threads: 16
timeout_seconds: 600

# Run against dev (default)
dbt run

# Run against prod explicitly
dbt run --target prod

# Use env var for CI/CD
export DBT_TARGET=prod
dbt run --target $DBT_TARGET

Why

profiles.yml separates credentials and connection details from the dbt project code, so the project repository contains no secrets. Per-developer dev datasets (dbt_alice, dbt_bob) prevent developers from overwriting each other's work or production tables.

Gotchas

profiles.yml lives in ~/.dbt/ by default — use --profiles-dir to override in CI/CD environments
Never commit profiles.yml to git — it contains database credentials
dbt_project.yml vars block is for static config; profiles.yml is for connection config — do not mix them
env_var() in profiles.yml raises a compilation error if the variable is not set — provide a default or fail explicitly

Code Snippets

Core dbt CLI commands for project initialization and model execution

# Initialize a new dbt project
dbt init my_project

# Test connection to configured profile
dbt debug

# Compile models without running (validates SQL)
dbt compile

# Run specific model and its downstream dependents
dbt run --select fct_orders+

# Run only changed models since last run (dbt Cloud / state comparison)
dbt run --select state:modified+

Context

Setting up a new dbt project or onboarding a developer to an existing one

Revisions (0)

No revisions yet.