Airflow vs Snowflake Tasks: Match the Orchestrator to Blast Radius

Last updated: August 2026

Every few months a client asks us to put Airflow in front of Snowflake, and about half the time the honest answer is: don't. Eleven SQL transformations run hourly, every input and output is a Snowflake table, and the plan is a scheduler with its own database, workers, upgrade path and on-call rotation, so it can send those statements back to the warehouse they came from.

The other half of the time it's the opposite. Someone has a nice task graph inside Snowflake and is now trying to make it wait on an SFTP drop, retry a flaky vendor API, and rebuild ninety days of history after a source correction. Tasks have no sensor for the first, no backfill for the third, and only an awkward external-function detour for the second. So: the rule we use, the criteria that decide it, both sides in code, and the cost question with arithmetic.

The verdict, before the detail

The rule is about where your dependencies live, not how complex the pipeline is. Complexity inside Snowflake is fine. Complexity that crosses the Snowflake boundary is where a general-purpose orchestrator earns its keep.

• Snowflake Tasks when every trigger and target is a Snowflake object. Streams tell you what changed, a root task owns the schedule, AFTER builds the graph, a finalizer handles cleanup and notification.
• Airflow when a step depends on something outside Snowflake (a file landing, an API, a Fivetran sync, a Spark job), when you replay date ranges on demand, or when people who aren't you need to see why last night failed and restart it.
• Both when the boundary-crossing part is small: Airflow owns the sensors and cross-system sequencing, then calls EXECUTE TASK or a dbt project so the work still runs inside Snowflake. This is where most of our clients end up.

The failure mode we see most isn't picking wrong. It's picking Airflow and treating it as an ETL engine, which is covered below and is worth more than the rest of this page combined.

Airflow vs Snowflake Tasks: the criteria that actually decide it

Ignore the marketing axes. These six decide it once you're past the demo.

One more axis if you're near real time: triggered tasks fire on stream content rather than the clock, at most every 30 seconds by default and down to 10 with USER_TASK_MINIMUM_TRIGGER_INTERVAL_IN_SECONDS, which Snowflake documents as a minimum of 10 and a default of 30. Airflow has no equivalent trigger, and its floor is whatever a scheduler loop plus a worker pickup costs you, which is not where it is designed to compete.

Cross-system dependencies are the real dividing line

A Snowflake task graph starts for exactly two reasons: a clock, or a WHEN predicate the root task can evaluate in SQL, usually SYSTEM$STREAM_HAS_DATA(). Everything after the root starts because its parents finished. There is no "wait until this S3 prefix has a _SUCCESS marker", because a task has no way to observe state outside the account and block on it.

The workaround is a polling task checking an external table or a control table someone else writes to. We've shipped it and it works, but be clear what you built: a polling loop with no timeout semantics and no way to tell "still waiting" from "upstream is broken". Sensors exist because that distinction matters at 3am.

The test: list every step and mark the ones whose start condition is a SQL boolean over Snowflake objects. If all of them are, Airflow is buying you a UI. If two or three aren't, you'll build a bad scheduler inside Snowflake to compensate.

What a Snowflake task graph looks like when you build it properly

Most graphs we inherit are a flat list of tasks chained with AFTER. The version worth writing has four things: a root task that owns the schedule, a stream predicate so the graph does nothing when there's nothing to do, an explicit overlap policy, and a finalizer.

-- Root task owns the schedule for the whole graph.
CREATE OR REPLACE TASK etl.load_root
  WAREHOUSE = etl_wh
  SCHEDULE = 'USING CRON 0 * * * * UTC'
  OVERLAP_POLICY = NO_OVERLAP
  SUSPEND_TASK_AFTER_NUM_FAILURES = 3
  USER_TASK_TIMEOUT_MS = 1800000
  ERROR_INTEGRATION = pipeline_alerts
  WHEN SYSTEM$STREAM_HAS_DATA('etl.orders_stream')
AS
  MERGE INTO analytics.dim_orders t
  USING (
      SELECT order_id, status, amount, order_date,
             metadata$action AS action
      FROM etl.orders_stream
      QUALIFY ROW_NUMBER() OVER (
                 PARTITION BY order_id
                 ORDER BY metadata$action DESC) = 1
  ) s
    ON t.order_id = s.order_id
  WHEN MATCHED AND s.action = 'DELETE' THEN DELETE
  WHEN MATCHED AND s.action = 'INSERT' THEN UPDATE SET
      t.status     = s.status,
      t.amount     = s.amount,
      t.updated_at = CURRENT_TIMESTAMP()
  WHEN NOT MATCHED AND s.action = 'INSERT' THEN INSERT
      (order_id, status, amount, order_date, updated_at)
  VALUES
      (s.order_id, s.status, s.amount, s.order_date, CURRENT_TIMESTAMP());

-- Child task: runs only after the root succeeds.
CREATE OR REPLACE TASK etl.refresh_marts
  WAREHOUSE = etl_wh
  AFTER etl.load_root
AS
  INSERT OVERWRITE INTO analytics.mart_daily_orders
  SELECT order_date, status, COUNT(*) AS orders, SUM(amount) AS amount
  FROM analytics.dim_orders
  GROUP BY 1, 2;

-- Finalizer: runs after every other task in the graph run ends,
-- whether the graph succeeded or failed.
CREATE OR REPLACE TASK etl.pipeline_finalizer
  WAREHOUSE = etl_wh
  FINALIZE = etl.load_root
AS
  CALL ops.record_and_notify_graph_result();

-- Resume every task in the graph at once, finalizer included.
-- The alternative is ALTER TASK ... RESUME on each child and the
-- finalizer first, then the root task last.
SELECT SYSTEM$TASK_DEPENDENTS_ENABLE('etl.load_root');

The QUALIFY is not decoration. Snowflake documents that updates to rows in the source object are represented as a pair of DELETE and INSERT records in the stream, with METADATA$ISUPDATE set to TRUE on both. So a single update already puts two rows with the same key into the stream. Join that raw stream to the target and the MERGE matches one target row twice, which Snowflake rejects by default because ERROR_ON_NONDETERMINISTIC_MERGE is TRUE. Collapsing to one row per key is the fix, and it's the Streams bug we get called in on most.

The finalizer is the piece people skip. Snowflake schedules it only when no other tasks are running or queued in the current task graph run, and it runs after the other tasks complete or fail to complete, which makes it the node to hang completion alerts on. Hanging that alert off the last child task is a silent trap: when an upstream task fails the children are never scheduled, so the alert you built for failures is the thing that doesn't fire. One caveat to know before you treat it as total coverage: if the root task run itself is skipped, the finalizer isn't started either, so a graph that quietly stops running still needs monitoring from outside.

The same pipeline as an Airflow DAG

Here's the equivalent DAG with the one thing Snowflake can't express bolted on the front: a wait for an external file. No transformation logic moved into Python. Airflow sequences and waits; Snowflake does the work.

import datetime

from airflow.sdk import DAG
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor

with DAG(
    dag_id="orders_hourly",
    start_date=datetime.datetime(2026, 1, 1),
    schedule="0 * * * *",
    catchup=False,
    max_active_runs=1,
    default_args={
        "retries": 2,
        "retry_delay": datetime.timedelta(minutes=5),
    },
    tags=["orders", "snowflake"],
) as dag:

    wait_for_extract = S3KeySensor(
        task_id="wait_for_extract",
        bucket_name="acme-landing",
        bucket_key="orders/{{ ds }}/_SUCCESS",
        aws_conn_id="aws_default",
        poke_interval=60,
        timeout=60 * 60 * 3,
        mode="reschedule",
    )

    merge_orders = SQLExecuteQueryOperator(
        task_id="merge_orders",
        conn_id="snowflake_default",
        sql="CALL etl.merge_orders()",
    )

    refresh_marts = SQLExecuteQueryOperator(
        task_id="refresh_marts",
        conn_id="snowflake_default",
        sql="CALL analytics.refresh_marts()",
    )

    wait_for_extract >> merge_orders >> refresh_marts

Two provider details worth getting right. For ordinary SQL the operator is SQLExecuteQueryOperator, which takes conn_id. It is not a Snowflake class: it lives in the common-sql provider and the Snowflake provider imports it, so the durable import path is airflow.providers.common.sql.operators.sql and you need both distributions installed. To submit several statements in one request, SnowflakeSqlApiOperator from apache-airflow-providers-snowflake goes through the Snowflake SQL API and takes snowflake_conn_id plus statement_count. Mixing those two connection argument names up causes a lot of "my DAG can't find the connection" tickets.

Set mode="reschedule" on long sensors, or deferrable=True, which S3KeySensor supports and which hands the waiting to the triggerer. In the default poke mode a sensor waiting three hours holds a worker slot for three hours, which is how one late vendor file stalls every other pipeline you own.

Serverless task vs warehouse task cost: where it crosses over

This is the sub-question people search separately and nobody answers with arithmetic. Two documented rates settle it.

• Warehouse tasks bill virtual warehouse credits. A standard XS warehouse is 1 credit per hour and each size doubles it (S is 2, M is 4, L is 8, XL is 16). Every start or resume consumes a minimum of one minute's worth of credits, and everything after that is per second, rounded up. Gen2 warehouses bill at a higher rate than standard, so read your own consumption table before reusing these numbers.
• Serverless tasks bill compute-hours, where one compute-hour is compute comparable to an XS warehouse for one hour, measured per second and rounded up. Serverless tasks carry a feature multiplier of 0.9, so the same compute costs 0.9 credits instead of 1.

So for identical compute serverless is 10% cheaper, with no one-minute floor and no idle tail, and a dedicated warehouse spun up for one task can never beat it. Take a task doing 40 seconds of XS-equivalent work every 15 minutes, 96 runs a day. Snowflake doesn't publish totals like these, so the figures below are worked from the two documented rates above rather than quoted from a price list:

Read the first two rows together. The gap isn't 10%, it's roughly 18x, and all of it is idle time waiting out the default 10-minute auto-suspend between short runs. That's the trap: a dedicated etl_wh nobody tuned, awake most of the day to do an hour of work.

The last row is the real crossover, and it has nothing to do with run length. A warehouse task beats a serverless task only when its billed warehouse seconds amortise across enough concurrent work that billed_warehouse_seconds / concurrent_task_seconds < 0.9. One task can never push that below 1, so you need at least two running side by side with almost no idle tail. Shared, densely packed warehouses win; dedicated per-pipeline warehouses lose.

The serverless side has its own trap. TARGET_COMPLETION_INTERVAL is not a throttle, it's a desired completion time, and Snowflake meets it by resizing the compute it puts behind the task. The tighter you set it, the larger the statement size it provisions and the more compute-hours the same work bills. Bound it with SERVERLESS_TASK_MIN_STATEMENT_SIZE and SERVERLESS_TASK_MAX_STATEMENT_SIZE, which run from XSMALL to XXLARGE, rather than finding the ceiling on an invoice. If your schedule is loose, Serverless Tasks Flex trades a wide execution window for a lower multiplier: Snowflake's consumption table lists Serverless Tasks at 0.9 and Serverless Tasks Flex at 0.5. It is a private preview feature, so check availability before you plan around it, and it requires SCHEDULING_MODE = 'FLEXIBLE', no WAREHOUSE, and both a schedule and a target completion interval of at least 60 minutes.

-- CREDITS_USED is typed VARCHAR in this view, so cast it explicitly
-- rather than relying on implicit coercion inside SUM().
SELECT
    task_name,
    ROUND(SUM(TO_DOUBLE(credits_used)), 3)        AS credits_30d,
    ROUND(SUM(TO_DOUBLE(credits_used)) / 30.0, 3) AS credits_per_day,
    COUNT(*)                                      AS metering_rows
FROM snowflake.account_usage.serverless_task_history
WHERE start_time >= DATEADD('day', -30, CURRENT_TIMESTAMP())
GROUP BY task_name
ORDER BY credits_30d DESC;

Each row is a metered START_TIME to END_TIME window, not a task run, so don't read the row count as a run count. Snowflake documents latency on this view of up to 180 minutes, which makes it useless for debugging a run from ten minutes ago. It also covers serverless only: warehouse tasks land in WAREHOUSE_METERING_HISTORY mixed in with everything else on that warehouse, which is why per-pipeline cost attribution is easier serverless.

Backfill: Snowflake has no catchup

Snowflake task scheduling only moves forward. If a task is still running when its next scheduled time arrives, that time is skipped. If a task is suspended, all future scheduled runs are cancelled, and resuming doesn't replay what was missed. EXECUTE TASK gives you one manual run, which is a testing tool, not a backfill mechanism.

For incremental Streams pipelines that's usually fine: the stream keeps its offset and the next successful run consumes what accumulated. It stops being fine when you reprocess history for reasons unrelated to change capture, like a source correction or a bug shipped three weeks ago. Then you write the loop yourself.

CREATE OR REPLACE PROCEDURE etl.rebuild_day(run_date DATE)
RETURNS STRING
LANGUAGE SQL
AS
$$
BEGIN
  DELETE FROM analytics.mart_daily_orders WHERE order_date = :run_date;

  INSERT INTO analytics.mart_daily_orders (order_date, status, orders, amount)
  SELECT order_date, status, COUNT(*), SUM(amount)
  FROM analytics.dim_orders
  WHERE order_date = :run_date
  GROUP BY 1, 2;

  RETURN 'rebuilt ' || run_date::STRING;
END;
$$;

-- Replay a range. Suspend the scheduled task first so a live run
-- does not collide with the rebuild.
ALTER TASK etl.load_root SUSPEND;

EXECUTE IMMEDIATE $$
DECLARE
  d DATE DEFAULT '2026-06-01'::DATE;
BEGIN
  WHILE (d <= '2026-06-30'::DATE) DO
    CALL etl.rebuild_day(:d);
    d := DATEADD('day', 1, d);
  END WHILE;
  RETURN 'backfill complete';
END;
$$;

ALTER TASK etl.load_root RESUME;

For a pipeline you backfill twice a year, that's the right answer. What you don't get is a record of which days were reprocessed, parallelism control, or the ability to hand the job to someone who won't run an anonymous block against production:

airflow backfill create \
  --dag-id orders_hourly \
  --from-date 2026-06-01 \
  --to-date 2026-06-30 \
  --reprocess-behavior failed \
  --max-active-runs 3

One correction to a widely repeated belief: in current Airflow, catchup is off by default. scheduler.catchup_by_default is False, so a DAG with a 2021 start date doesn't stampede a thousand runs when you unpause it. That removes the main historical argument against Airflow in front of a warehouse.

Failure handling, retries, and who gets paged

Both sides retry. The difference is what happens once retries are exhausted.

• Snowflake: TASK_AUTO_RETRY_ATTEMPTS retries a failed task graph from the task that failed rather than retrying one task in isolation, it defaults to 0, and it has to be set on the root task or Snowflake returns an error. SUSPEND_TASK_AFTER_NUM_FAILURES suspends a standalone task, or the root task of a graph, after N consecutive failures or timeouts. USER_TASK_TIMEOUT_MS stops a hung statement, and ERROR_INTEGRATION pushes failure notifications to Amazon SNS, Azure Event Grid or Google Pub/Sub.
• Airflow: per-task retries and retry_delay, exponential backoff, on_failure_callback, and since 3.3 pluggable retry policies, so you can retry based on the exception rather than a fixed count.
• The real gap: Airflow shows a failed task in a UI with logs, previous attempts and a clear button. Snowflake shows rows in TASK_HISTORY and a graph run view in Snowsight, which is good now, but whoever clears it needs account access and enough SQL to know what to re-run.

The gotcha worth writing on the wall: SUSPEND_TASK_AFTER_NUM_FAILURES suspends the task, and a suspended task cancels all its future scheduled runs. If your alerting is itself a scheduled task, the mechanism protecting the pipeline can quietly disable your alarm. Put notifications in the finalizer, which runs regardless of outcome, and monitor the tasks from outside.

Airflow is a scheduler, not an ETL engine

If you take one thing from this page, take this. The most expensive mistake in Airflow projects isn't choosing Airflow. It's writing tasks that pull data out of Snowflake into a worker with pandas, transform it in memory, and write it back. We've inherited that more than once and it ends the same way: workers sized for the biggest table anyone touches, a memory error at 2am the night volumes spiked, and a Snowflake bill that didn't fall.

The transformation belongs where the data is. Airflow's job is to decide when and in what order, and to know whether it worked.

• Push SQL down. SQLExecuteQueryOperator calling a stored procedure or a dbt model, not a Python function holding a dataframe.
• Defer anything that waits. SnowflakeSqlApiOperator and the AWS sensors take deferrable=True, which hands the polling to the triggerer and releases the worker slot. Know the exception: SQLExecuteQueryOperator has no deferrable mode, so a long-running statement submitted through it occupies a worker for its whole duration. If that matters, run it through the SQL API operator instead.
• Keep state out of the DAG file. Watermarks and cursors used to live in an XCom hack or a side table. Airflow 3.3 added a first-class state store with task_state_store and asset_state_store, backed by the metadata database and configurable under [state_store]. If you kept a Snowflake control table purely because Airflow had nowhere sensible for a watermark, that reason is gone.

The test: if you scaled your workers down to the smallest instance your provider offers, would any pipeline break? If yes, you're running an ETL engine and paying for a scheduler.

What shifted in 2026, on both sides

On the Airflow side, 3.3.0 landed on 6 July 2026, headlined by the state store above. With it came pluggable retry policies, a major expansion of asset partitioning, and a Language Task SDK that lets task bodies be written in Java or Go while orchestration stays in Python. Treat that last one as a signal, not something to build on: Airflow labels it experimental in this release and warns the APIs and wire protocol may change between versions.

On the Snowflake side, there's less left to orchestrate at all. dbt Projects on Snowflake went GA on 6 November 2025, so a dbt project can be a schema-level object in your account and a task can simply run it with EXECUTE DBT PROJECT. The DAG inside the project is resolved by dbt, not your orchestrator, so a graph that used to be thirty Airflow tasks collapses to two:

-- The task must live in the same database and schema as the dbt
-- project object, and it must use a user-managed warehouse:
-- serverless tasks cannot run EXECUTE DBT PROJECT.
CREATE OR ALTER TASK analytics.transform.run_dbt_prod
  WAREHOUSE = transform_wh
  SCHEDULE = '6 hours'
  SUSPEND_TASK_AFTER_NUM_FAILURES = 2
  ERROR_INTEGRATION = pipeline_alerts
AS
  EXECUTE DBT PROJECT analytics.transform.core_models
    ARGS = 'run --target prod';

CREATE OR ALTER TASK analytics.transform.test_dbt_prod
  WAREHOUSE = transform_wh
  AFTER analytics.transform.run_dbt_prod
AS
  EXECUTE DBT PROJECT analytics.transform.core_models
    ARGS = 'test --target prod';

ALTER TASK analytics.transform.test_dbt_prod RESUME;
ALTER TASK analytics.transform.run_dbt_prod RESUME;

Alongside it, dynamic tables gained a SCHEDULER attribute that reached GA on 26 March 2026: SCHEDULER = DISABLE takes a table out of automatic refresh so you can drive it with ALTER DYNAMIC TABLE ... REFRESH. It's a dynamic table attribute, not a task one, but it closes the last gap where declarative tables couldn't be folded into a schedule you control.

How we make the call on client projects

The sequence takes about twenty minutes with the team:

• List every start condition. Any that isn't a SQL boolean over Snowflake objects is a vote for Airflow. Zero of them is a strong vote against it.
• Ask how often you replay history. Twice a year: write the loop. Monthly, or by someone who isn't a data engineer: you want real backfill.
• Ask who is on call. One engineer who lives in Snowsight, and Tasks are fine. A rotation including people without deep SQL, and the UI is worth real money.
• Price both. Serverless tasks at 0.9 credits per compute-hour, against a managed Airflow instance plus the same Snowflake compute anyway.
• If Airflow wins, keep it thin. Sensors, sequencing, backfill and alerting in Airflow. Every transformation in Snowflake, invoked with EXECUTE TASK, a procedure, or a dbt project.

You aren't really picking a winner in Airflow vs Snowflake Tasks. You're deciding how much of the pipeline lives outside the database, and keeping that number as small as your dependencies permit.