Dagster Assets Dependencies and Data-Aware Orchestration

Published by Programming Line in Data Engineering in Depth. Last updated .

Purpose: orchestrate data by asset, not by task

Dagster assets let a data engineering team describe the things the platform owns: tables, files, model outputs, dashboards, or intermediate datasets. A dependency edge means one asset is computed from another asset. Data-aware orchestration is the behavior that follows from that graph: Dagster can decide what to run, what is stale, what failed, and what can be skipped by reasoning about data products instead of only executing a fixed list of tasks.

The outcome of this lesson is practical: you should be able to model a small warehouse flow as Dagster assets, read the dependency graph, choose explicit or inferred dependencies, select subsets for a job, and troubleshoot why an asset did or did not materialize. In a data engineering course, this is the bridge between transformation code and operations. SQL, Python, and loaders create data; an asset graph gives those outputs names, lineage, schedules, checks, and incremental repair paths.

How Dagster builds the asset graph

An asset definition combines a key, a compute function, dependency metadata, optional partitions, resources, config, and checks. The asset key is the stable address of the data product. A simple key such as order_totals names one asset. A path key such as ["warehouse", "customers"] names a grouped asset and usually maps well to schemas, folders, or domains.

Dagster infers many dependencies from function parameters. If an asset function named order_totals has a parameter named raw_orders, Dagster treats raw_orders as an upstream asset input. During materialization, Dagster topologically sorts the graph so upstream assets run before downstream assets. The downstream function receives the upstream value through the active I/O manager. In local examples the value may be an in-memory Python object. In production, an I/O manager often writes and reads tables, files, object-store paths, or dataframe formats.

Materialization is Dagster’s record that an asset was produced. The event log stores which asset key was materialized, run identifiers, metadata, partitions, checks, and failure information. Dagster uses those events to render lineage and to reason about freshness and staleness. If raw_orders changes after order_totals was last materialized, the downstream asset can be considered stale because the latest known upstream data is newer than the downstream output.

This is different from a task-only DAG. In a task DAG, a node often means “run this operation.” In an asset graph, a node means “this data product exists and can be materialized.” The distinction matters when only one table is broken. You can select that table and its needed parents rather than rerunning a whole workflow because the graph has data-level addresses.

API anatomy

The core decorator is @asset. The function name becomes the asset key unless you override it. Parameters normally become asset inputs. AssetIn lets you map a friendly parameter name to a different asset key, set metadata, or describe partition mapping. deps declares dependencies that do not need to be passed as Python arguments, which is common when a SQL statement reads from a table by name instead of consuming an in-process value.

Definitions is the object that collects assets, jobs, schedules, sensors, checks, and resources for a Dagster code location. define_asset_job creates a job from an asset selection. AssetSelection can include exact assets, groups, upstream dependencies, downstream dependencies, or checks. This makes jobs projections of the same asset graph rather than separate copies of orchestration logic.

Resources provide clients and configuration to assets: database connections, object-store clients, API clients, dbt projects, Spark sessions, or custom services. I/O managers control how values move between assets. Partitions split an asset into addressable slices, such as daily sales for 2026-09-07. Partition-aware dependencies let a downstream daily asset consume the matching upstream daily partition, a shifted window, or a wider range.

Example 1: inferred dependencies

The smallest useful asset graph has a raw input and a derived output. In the code below, order_totals depends on raw_orders because its argument has the same name as the upstream asset.

from dagster import AssetExecutionContext, Definitions, asset, materialize

@asset
def raw_orders() -> list[dict]:
    return [
        {"order_id": 1, "customer_id": "C-1", "amount": 100.0},
        {"order_id": 2, "customer_id": "C-2", "amount": 50.0},
    ]

@asset
def order_totals(raw_orders: list[dict]) -> dict[str, float]:
    totals: dict[str, float] = {}
    for row in raw_orders:
        totals[row["customer_id"]] = totals.get(row["customer_id"], 0.0) + row["amount"]
    return totals

if __name__ == "__main__":
    result = materialize([raw_orders, order_totals])
    assert result.success
    print("materialized raw_orders -> order_totals")

When executed in an environment with Dagster installed, the materialization engine runs raw_orders first, passes its returned list to order_totals, records both materializations, and prints materialized raw_orders -> order_totals. If order_totals raises an exception, the event log still shows that raw_orders succeeded in that run, which helps you retry only the failed downstream work after fixing the transform.

Example 2: explicit input mapping

Real graphs rarely use perfect Python names. A warehouse table may be keyed as warehouse/customers, while the downstream function wants a parameter named customer_status. AssetIn separates the function signature from the upstream asset key.

from dagster import AssetIn, asset

@asset(key_prefix="warehouse")
def customers() -> dict[str, str]:
    return {"C-1": "active", "C-2": "inactive"}

@asset(
    ins={
        "orders_by_customer": AssetIn(key="order_totals"),
        "customer_status": AssetIn(key=["warehouse", "customers"]),
    }
)
def active_customer_revenue(
    orders_by_customer: dict[str, float],
    customer_status: dict[str, str],
) -> float:
    return sum(
        amount
        for customer_id, amount in orders_by_customer.items()
        if customer_status.get(customer_id) == "active"
    )

This asset has two parents: order_totals and warehouse/customers. The expected computed value is 100.0 because customer C-1 is active and customer C-2 is inactive. The important behavior is that the graph edge follows the declared key, not the parameter name. This prevents accidental renames from changing lineage and lets teams use domain-oriented asset keys while keeping readable local code.

Example 3: selecting the right slice of the graph

Asset jobs are usually defined by selecting assets from the graph. A revenue job can target a final asset and include its upstream dependencies.

from dagster import AssetSelection, define_asset_job

nightly_revenue_job = define_asset_job(
    name="nightly_revenue_job",
    selection=AssetSelection.assets("active_customer_revenue").upstream(),
)

print(nightly_revenue_job.name)

The deterministic output is nightly_revenue_job. In a Dagster deployment, the selection means the job includes active_customer_revenue and the assets it needs upstream. The exact resolved set depends on the asset definitions loaded with the job. This pattern is useful when several jobs share the same graph: a nightly job might materialize a full business domain, while an hourly job materializes only latency-sensitive assets.

Example 4: checks as data-aware gates

Asset checks attach validation results to an asset key. They do not replace tests in source control; they record whether materialized data satisfied a runtime expectation.

from dagster import AssetCheckResult, asset, asset_check

@asset
def order_totals() -> dict[str, float]:
    return {"C-1": 100.0, "C-2": 50.0}

@asset_check(asset=order_totals)
def no_negative_customer_revenue(order_totals: dict[str, float]) -> AssetCheckResult:
    negatives = [customer_id for customer_id, amount in order_totals.items() if amount < 0]
    return AssetCheckResult(
        passed=not negatives,
        metadata={"negative_customer_count": len(negatives)},
    )

check_result = no_negative_customer_revenue(order_totals())
print(check_result.passed)

The output is True. If a negative amount appears, the check result changes to failed and records the count in metadata. Downstream automation can use these results differently depending on risk: a dashboard refresh might continue with a warning, while a financial export might stop until the asset is corrected.

Design choices and trade-offs

Prefer inferred parameter dependencies when Python names are stable and the graph is small. They keep code compact and easy to read. Prefer AssetIn when upstream keys are namespaced, when refactoring parameter names, or when one function consumes several similarly shaped inputs. Prefer deps for side-effect dependencies, especially SQL assets that read upstream tables inside a query and do not need the upstream data as Python objects.

Asset granularity is the main modeling decision. A table-level asset graph is easy to operate and aligns with warehouse lineage. A column-level graph is usually too noisy. A broad “all marts” asset hides failure boundaries and makes reruns expensive. A good rule is to make an asset for each independently materialized data product with a clear owner and consumer.

Partitions improve backfills and retries but add complexity. Daily partitions are natural for event data and snapshots. Unpartitioned assets are better for small reference tables or outputs that are always rebuilt as a whole. Multi-dimensional partitions can model date by region or date by tenant, but they require careful selection rules so a small correction does not trigger unnecessary recomputation.

Failure modes and troubleshooting

Symptom: a downstream asset is missing from a job. The usual cause is an asset selection that names only the final asset without including upstream dependencies, or definitions that were not loaded into the same code location. Inspect the Dagster asset graph and resolved job selection. Correct it by using AssetSelection.assets("name").upstream(), selecting a group, or ensuring all relevant asset modules are included in Definitions.

Symptom: Dagster reports an unresolved asset input. The cause is often a parameter name that does not match any asset key, or an AssetIn key path that differs from the actual asset key. Compare the upstream asset key shown in the UI with the function input mapping. Correct the spelling, add AssetIn, or rename the asset intentionally with a migration plan for consumers.

Symptom: a materialization succeeds but downstream data is stale. Common causes include reading an upstream table directly without declaring deps, using an external table that Dagster does not observe, or materializing only a subset of partitions. Diagnose by checking lineage edges and the latest materialization timestamps for the relevant partition. Correct it by declaring the dependency, adding an observable source asset for external data, or backfilling the missing partitions.

Symptom: local examples pass but production runs fail to load inputs. The likely cause is an I/O manager mismatch. In-memory passing hides serialization and storage issues. Check the configured I/O manager, object paths, table names, credentials, and data formats. Correct it by configuring a production I/O manager consistently across upstream and downstream assets and by adding metadata that records the physical destination.

Security, reliability, and performance implications

Asset dependencies can expose sensitive lineage. If an asset represents restricted customer data, avoid naming keys or metadata in a way that leaks protected details to broad UI audiences. Use resources to centralize credentials rather than embedding connection strings in asset functions. For reliability, declare every real data dependency. Hidden reads make stale data harder to detect and make backfills incomplete. For performance, use selections and partitions to limit recomputation. Do not pass large production datasets as Python return values unless the I/O manager is designed for that pattern; materialize to durable storage and pass references through managed I/O.

Hands-on lab: build and inspect a small asset graph

Prerequisites: Python, a virtual environment, and Dagster installed in that environment. Use a temporary directory so cleanup is simple.

  1. Create a file named assets_lab.py containing the assets from examples 1 and 2, plus a Definitions object that includes raw_orders, order_totals, customers, and active_customer_revenue.
  2. Run the file directly and verify that materialization succeeds for the first two assets.
  3. Start Dagster’s development UI for the module, open the asset graph, and confirm that order_totals has an upstream edge from raw_orders.
  4. Add the mapped customer dependency from example 2, reload definitions, and confirm that active_customer_revenue has two parents.
  5. Define nightly_revenue_job with the upstream selection from example 3 and launch it from the UI or command line.
  6. Verification: the graph should show four assets, the revenue asset should resolve to 100.0 with the sample data, and the latest materialization timestamps should update for selected assets.
  7. Cleanup: stop the dev server, deactivate the virtual environment, and remove the temporary lab directory. If you used a persistent Dagster home, remove only the lab instance data you created.

Assessment exercises

  1. You rename a function parameter from raw_orders to orders and Dagster can no longer resolve the input. Write the smallest code change that preserves the original upstream dependency.
  2. A SQL asset reads warehouse.customers inside the query but has no Python argument for it. Explain whether to use an inferred dependency, AssetIn, or deps, and why.
  3. A team wants one asset for an entire reporting schema containing thirty tables. Identify two operational problems this creates and propose a better asset boundary.
  4. A downstream daily partition is stale even though today’s upstream partition materialized. List the Dagster views or event details you would inspect before rerunning anything.
  5. Design a check for active_customer_revenue that would catch an obviously invalid output without blocking harmless late-arriving raw orders.

Summary

Dagster asset dependencies turn orchestration into a graph of named data products. Function parameters, AssetIn, deps, selections, partitions, I/O managers, and checks each describe a different part of how data is produced and trusted. The practical advantage is targeted operation: you can materialize, backfill, validate, and repair the specific assets affected by a change instead of treating the pipeline as an opaque sequence of tasks.