DOCS / PLATFORM

Actual cost

Everything else in Finowl is calculated from a bundle. This is the other half: what you were billed, attributed back to the product that spent it. It needs two connections and one tag.

Why two sources

Databricks bills the platform units your workloads consume. Your cloud provider bills the machines those units run on, plus their storage and data transfer. Neither is complete alone, and each looks complete when read alone.

In a case we measured, the infrastructure half was the larger of the two: a virtual machine line of €1.94 against €1.54 of platform units. A figure from one source understates a product by roughly half.

COMPONENT	OWNING SOURCE	COVERS
Platform units	Databricks system tables	DBU consumption, classic and serverless
Infrastructure	Azure Cost Management	VMs, managed disks, NICs, public IPs, NAT gateways
Storage	Azure Cost Management	Storage attached to running compute
Data transfer	Azure Cost Management	Bandwidth and egress

Each component has exactly one owning source. Databricks charges appear in Azure Cost Management too, so the Databricks service line is dropped on ingestion. Without that, every product would report its platform units twice.

Setting it up

1

Connect an Azure subscription

Cost is queried per subscription. Azure reserves the term “billing account” for the enrolment above subscriptions, which is not what you connect here. Add one connection per subscription you are billed through, in Settings → Integrations.

Create a service principal and grant it two roles on the subscription: Cost Management Reader to read what was billed, and Reader to see which Databricks workspace each cost came from. Then supply the subscription id, tenant id, client id and client secret.

NOTE

Cost Management Reader alone is not enough. It grants no read on resources, and Azure answers a request it cannot fulfil with an empty list rather than an error — so cost still arrives, in full, while every environment reads as unknown. Adding Reader is what resolves them.

The connection is tested before it is saved. A missing role produces an error that says so rather than a generic failure — the two have different fixes, and a credentials error sends you to the wrong place.

NOTE

If a subscription covers exactly one environment, say which — it labels spend we cannot trace to a workspace. If it hosts several, choose hosts several environmentsand each workspace’s spend is labeled from the workspace you mapped it to. Declaring one environment for a subscription that hosts several is what makes development spend read as production. Two subscriptions may share an environment; their spend is summed rather than one overwriting the other.

2

Nominate a usage-query workspace

Databricks system tables are account-global: they hold usage from every workspace in the account, but can only be read from a workspace enabled for Unity Catalog.

So you nominate one workspace and its SQL warehouse, and that single nomination covers the whole account — including workspaces that are not Unity Catalog enabled themselves.

The identity behind the connection needs these grants, run from that workspace:

GRANT USE CATALOG ON CATALOG system TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA system.billing TO `<principal>`;
GRANT SELECT ON TABLE system.billing.usage TO `<principal>`;
GRANT SELECT ON TABLE system.billing.list_prices TO `<principal>`;

list_prices is not optional. The usage table has no cost column — cost is the quantity consumed multiplied by the price valid at the time, so both tables are needed to produce a figure.

3

Tag your compute

Spend is attributed by an attribution_id tag carrying the product id from finowl.yml.

databricks.yml

resources:
  jobs:
    daily_ingest:
      job_clusters:
        - job_cluster_key: main
          new_cluster:
            node_type_id: Standard_DS3_v2
            num_workers: 4
            custom_tags:
              attribution_id: 6f1c9a2e-4b13-4d0e-9f1a-2c7d8e5b0a34

For classic compute the tag goes in the cluster block's custom_tags. Databricks propagates workspace, pool and cluster tags onto the machines it creates; a job-level tag never reaches them, so on its own it would prove nothing about the infrastructure the job generates.

Put the same tag on the job or pipeline itself, in its own tags. Serverless jobs and pipelines have no cluster to tag, and their tags never appear on the billing records. But every billing record names the job or pipeline it came from, so Finowl reads the tag from the job's definition in system.lakeflow. No usage policy is needed. The service principal needs SELECT on system.lakeflow.jobs and system.lakeflow.pipelines for this; without it, only serverless spend goes unattributed, and the connection panel names the missing grant.

One rule keeps attribution complete

Attribution is enforced by one rule created with every organization. It cannot be edited or turned off: the product identity tag is required on job, pipeline, job_cluster, pipeline_cluster and warehouse. The cluster tag is what reaches the cloud bill; the job and pipeline tag is what attributes serverless spend.

IMPORTANT

Every cost figure on a product page is traced through this rule. An organization able to weaken it would be switching off its own reporting — the page would still show a total, built from a fraction of the money, with nothing to indicate it.

Untagged spend is never guessed at or apportioned. It is left unattributed, and the product page says which half is missing.

When figures appear

Retrieval runs once a day and re-reads from the first of the previous month through today, replacing what it holds. Both sources revise figures for days afterwards, so a window that never looked back would freeze whatever a day happened to cost the first time it was read.

Every figure carries an as-of date, and it is the last date the source actually covered — not today. The most recent day or two are always partial. A figure presented as current when it is not reads as an unexplained dip, and somebody investigates a drop that never happened.

Storage and data transfer in the CLI

finowl breakdown shows measured storage and data transfer below its calculated tables, labelled with the period they cover. They are not added into the estimated total — a measurement folded into a projection produces a figure that is neither.

Nothing measured yet is stated rather than shown as zero. Nothing measured and nothing spent are different facts.

Partial setups

WHAT IS MISSING	WHAT YOU STILL GET
No Unity Catalog workspace anywhere in the account	Classic infrastructure from Cost Management. Platform units cannot be read at all, and the product page says so.
No Azure connection	Platform units only — roughly half the true figure. Connect the subscription before drawing conclusions from it.
Compute not tagged	Nothing for the untagged resources. Their spend is real and unattributed, which the page reports rather than hides.

Reading the result

Budget against bill — the operational question. Are we overspending this month?
Baseline against bill — the planning question. Did we estimate this correctly? Feeds Learn.
Projection against the bill's compute — the modelling question. A gap here usually means the bundle does not describe what actually runs: a workload word that is wrong, or a schedule that fires more often than declared.

Keeping the three apart matters. A product over budget because the estimate was wrong needs a different conversation from one over budget because a job runs four times more often than the bundle says.

PREVIOUS

The web app

NEXT

History and audit trail