Databricks Cost Control: System Tables, Policies and Serverless Modes

Last updated: October 2026

A Databricks bill rarely grows because of one large decision. It grows through clusters left running, scheduled jobs on interactive compute, SQL warehouses created with default settings, and maintenance work that nobody remembers switching on. Each of those has a specific control, and most of the controls are configuration settings.

This guide covers how to measure spend from Databricks' system tables and the controls that reduce it, as documented in October 2026. Rates differ by cloud, region and plan, so it does not quote per-DBU prices and instead shows how to calculate your own from the list prices table. For tuning the Spark work itself, see our guide to Databricks Spark optimisation.

Measure spend with the billing system tables

system.billing.usage holds one row per usage record, with the quantity consumed, the SKU, the product that generated it (such as JOBS, SQL, ALL_PURPOSE or DLT), any custom tags, and metadata such as the job ID, warehouse ID or cluster ID. New records typically appear within 12 hours. system.billing.list_prices adds a row each time a SKU's price changes, so a join has to match the price period as well as the SKU, which is how Databricks' own cost queries are written.

-- List-price cost by team tag and product, last 30 days (Databricks SQL)
SELECT
  u.custom_tags['team']                     AS team,
  u.billing_origin_product                  AS product,
  SUM(u.usage_quantity)                     AS usage_units,
  SUM(u.usage_quantity * p.pricing.default) AS list_cost
FROM system.billing.usage AS u
JOIN system.billing.list_prices AS p
  ON  u.cloud = p.cloud
  AND u.sku_name = p.sku_name
  AND u.usage_start_time >= p.price_start_time
  AND (u.usage_end_time <= p.price_end_time OR p.price_end_time IS NULL)
WHERE u.usage_date >= CURRENT_DATE() - INTERVAL 30 DAY
GROUP BY ALL
ORDER BY list_cost DESC;

Three details decide whether these numbers can be trusted:

• They are list prices. Databricks' cost examples use pricing.default, while the list prices page describes pricing.effective_list as the price used for cost calculation. Budgets use the same list-price basis and exclude credits and negotiated discounts, so none of these will match an invoice exactly.
• Usage rows are corrected after the fact. A wrong record receives a RETRACTION row with a negative quantity and then a RESTATEMENT, so sum every record type; filtering to ORIGINAL rows ignores the corrections.
• Attribution has gaps. usage_metadata.job_id is populated for serverless jobs and job compute, not for all-purpose compute, and job names have only been recorded since September 2025.

Run scheduled jobs on job compute or serverless

Scheduled jobs on all-purpose clusters are a common source of hidden cost. They bill under the all-purpose product, a separate SKU from job compute, and because no job ID is recorded for them, Databricks' per-job cost queries leave them out entirely. Moving them to job compute or serverless makes each job visible in the usage table; to compare rates for your cloud and region, query list_prices for the two SKUs.

Policy families help here. Databricks provides Personal Compute, Shared Compute, Power User Compute and Job Compute families as templates. The Personal Compute policy is available to every workspace user by default unless it is disabled at account level, which means any user can create single-node compute. The other three are available only to workspace admins by default.

Compute policies for classic compute

Compute policies are where spending limits on classic compute are actually enforced. A policy can fix a value and hide it, set a range, or limit choices to an allowlist, and it applies to every cluster created with it. The example below fixes auto-termination at 30 minutes, caps autoscaling and hourly DBUs, limits node types, requires a team tag from a fixed list, and uses spot instances with on-demand fallback.

{
  "autotermination_minutes": { "type": "fixed", "value": 30, "hidden": true },
  "autoscale.max_workers": { "type": "range", "maxValue": 10 },
  "node_type_id": {
    "type": "allowlist",
    "values": ["i3.xlarge", "i3.2xlarge"],
    "defaultValue": "i3.xlarge"
  },
  "custom_tags.team": { "type": "allowlist", "values": ["finance", "marketing", "platform"] },
  "dbus_per_hour": { "type": "range", "maxValue": 50 },
  "aws_attributes.availability": { "type": "fixed", "value": "SPOT_WITH_FALLBACK" }
}

dbus_per_hour and cluster_type are virtual attributes, so a policy can cap the hourly DBU rate directly, or restrict a policy to job compute by fixing cluster_type to job. On spot instances, the driver and the first instance are always on-demand, and fleet instance types such as m-fleet.xlarge pick whichever instance type of that size has the best spot capacity and price at launch. With Premium or above, optimised autoscaling scales down after 40 seconds of underuse on job compute and 150 seconds on all-purpose compute.

Tags need care to be useful. On AWS, instances in clusters created from a pool inherit the workspace and pool tags but not the cluster tags, so put cost tags on the pool. Tag changes apply only after a restart or pool expansion, each resource takes at most 20 tags, and keys and values cannot contain spaces or slashes.

Serverless: standard mode and usage policies

Serverless jobs and pipelines run in performance-optimised mode by default, which starts quickly and suits interactive or time-sensitive work. Standard mode uses less compute and takes about 4 to 6 minutes to start, and Databricks' best-practices page says it can reduce costs by up to 70% compared with performance-optimised mode. It uses the same SKU and consumes fewer DBUs. Standard mode is available for Lakeflow Jobs and Lakeflow pipelines but not for notebooks, and since 6 July 2026 it also covers materialized view and streaming table refreshes in jobs. The mode used is recorded in product_features.performance_target in the usage table.

Most nightly batch jobs can tolerate a few minutes of start-up time, so standard mode is the first setting we check on a serverless estate. One related setting matters for cost and correctness: serverless auto-optimisation can retry a task, and retries can be turned off for a task that must run at most once.

Classic compute is tagged through policies, but serverless work is tagged through serverless usage policies, the feature previously called budget policies, which is still in Public Preview. A policy attaches custom tags to the serverless usage of the users assigned to it. Existing jobs are not tagged automatically, pipelines triggered by a job do not inherit the job's policy, and the usage table's budget_policy_id column is deprecated in favour of usage_policy_id, which affects older dashboards.

SQL warehouses

Databricks offers serverless, pro and classic SQL warehouses. Serverless starts in about 2 to 6 seconds and runs in the Databricks account. Pro and classic take about 4 minutes to start and run in your own cloud account, so they also incur your cloud provider's instance charges. In the UI a new warehouse is serverless where that is supported, but a warehouse created through the API with default parameters is classic, and Databricks recommends setting the type explicitly.

Auto stop defaults differ by type. Pro and classic warehouses stop after 45 minutes of inactivity by default, with a minimum of 10. Serverless warehouses default to 10 minutes, can go down to 5 in the UI and to 1 minute through the API. Databricks notes that idle warehouses keep accumulating DBU and cloud instance charges until they stop, and the default cluster size is X-Large, so it is worth starting smaller and sizing up from query profiles. For concurrency, Databricks recommends one cluster for every 10 concurrent queries.

Photon: check that it pays

Photon consumes DBUs at a different rate from the same instance type on the standard runtime. Databricks says it gives the greatest benefit on longer-running queries over large datasets, and that queries finishing in under two seconds see no meaningful improvement. It falls back to the Spark engine for UDFs, RDD and Dataset APIs and stateful streaming, so a job dominated by Python UDFs can pay the Photon rate without much of the speed-up.

Photon is the default for SQL warehouses and serverless compute, and it is on by default for classic compute created in the UI; through the Clusters or Jobs API you set runtime_engine to PHOTON. The query profile shows the share of task time spent in Photon, and product_features.is_photon in the usage table shows which usage was billed with it.

Budgets, predictive optimization and other 2026 changes

Budgets became generally available on 6 July 2026. A budget tracks spend in US dollars at list price across the account, or filtered by workspace or custom tag, with up to four monthly thresholds and email alerts. Budgets alert; they do not stop usage, and alert emails can lag usage by up to 24 hours. The one blocking option applies to Unity Gateway requests, and Databricks warns against relying on it as an absolute spending cap. Hard limits therefore come from the controls above.

Predictive optimization can add spending that appears without anyone creating a job. It runs OPTIMIZE, VACUUM and ANALYZE on Unity Catalog managed tables, is enabled by default for accounts created on or after 11 November 2024, and was being rolled out to existing accounts with completion expected by August 2026. It is billed on a serverless jobs SKU, its operations and costs are recorded in system.storage.predictive_optimization_operations_history, and it can be disabled per catalog, schema or table. Whether to keep it on is a judgement for each team, but it should be visible in the cost reports.

Genie products also moved to pay-as-you-go pricing on 8 July 2026, with 150 free DBUs of LLM usage each month, so teams using Genie should include it in their cost dashboards.