Core concepts

Accounts and organizations

An account is your company. It contains one or more organizations — typically one per department, domain or environment boundary. Every record belongs to exactly one organization, and no query crosses that line.

Policies can be written at either level. An account policy binds every organization beneath it; an organization policy refines things locally. At evaluation time Finowl merges the two, and the account rule wins where they overlap.

Data product

The unit of governance: a pipeline, a set of notebooks, a lakehouse project. A data product carries a budget, tags, and a lifecycle status — draft, development, staging, production, archived.

Products are created by registration, never by a report. A report carrying an identifier the platform does not know is rejected rather than quietly creating something. That is deliberate: a product that appears because of a typo has no owner and no budget, and nobody notices it exists.

Identity, name and label

Three different things, and confusing them is how run histories get mixed.

Because identity is an opaque id rather than a repository URL, moving a repository, restructuring a monorepo or migrating to another deployment tool never loses a product's history.

Deployment

One environment of a data product: development, staging or production. It comes from the --target the CLI was run with. Each deployment keeps its own run history, its own projected cost and its own actual month-to-date spend.

A deployment is created by the first run that reports from it. That first run also establishes the diff baseline: there is nothing to compare it against, so it becomes the comparison for the next one.

Workload words

Finowl cannot know how long your job runs. Databricks does not publish it before the job exists, and a pull request that has never been deployed has no history to read. So the person who wrote the job answers a single question per compute: roughly how much work runs on it per run.

The word times the compute; the schedule in the IaC sets how often it runs; the cluster shape sets what an hour costs. Words are recorded per target and never inherited between environments — see finowl.yml reference.

CLI run

One execution of the CLI against one deployment. A run holds the cost breakdown, the diff against the previous state, and every rule result. Runs come in two kinds, and the difference matters.

The CLI detects which it is. An engineer running finowl breakdown on a laptop cannot accidentally move the number the business is reading.

Cost estimate and the baseline

An estimate is a pre-development projection, made inside the data product it belongs to. Most come from the estimator, which holds the conversation and the assumptions alongside the figure. A number you already hold — from a spreadsheet or a signed statement of work — can be entered by hand and behaves identically from there.

Every estimate carries two figures. The P50 is the midpoint: as likely to be exceeded as not. The P80 is sufficient in about 80% of the modelled outcomes, and it is what Finowl proposes as the budget, because a budget set at the midpoint is under-funded half the time. The P80 is the P50 plus a published margin — 25% by default — and both figures freeze with the estimate, so changing that default later cannot move a budget somebody already agreed to.

A product has exactly one baseline: the estimate that was agreed for it. Committing a new estimate supersedes the previous one, which is kept, marked superseded, and carries the reason it was replaced.

Commitment

Committing is the moment a budget exists. One act freezes the estimate as the agreement, records who agreed it and when, and sets the budget — so there is never a window where the estimate is agreed and the money is not.

Two things can change afterwards, and they are different acts:

• Changing the budget moves the money and leaves the estimate alone. The baseline keeps its version, so nothing tells your engineers their code is out of date.
• Re-estimating produces a new estimate. Committing it supersedes the previous one and requires a reason, because that reason is the only record of why the agreement moved.

Rules and policies

A rule is one check with one severity. A policy is a named group of rules targeted at one or more environments. A rule in no policy never runs, whatever it says.

cost:
  currency: USD
  thresholds:
    - name: Budget exceeding
      type: absolute            # the ceiling is this product's budget
      value: 400
      action: block
    - name: No large jumps
      type: increase_absolute   # measures the change, not the total
      value: 50
      action: block

governance:
  rules:
    - name: Team tag on compute
      type: required_tag
      key: team
      scope: [job_cluster, pipeline_cluster]
      enforcement: allowlist
      allowed_values: [data-engineering, ml, platform]
      action: block

The CLI never holds this file. It asks the platform for the merged, deduplicated result for this product and this environment, so a rule change takes effect on the next pipeline run with nothing to redeploy. Full reference in Policies and rules.

Severity

Every rule carries an action. block makes finowl guard exit non-zero, which fails the pipeline. warn reports the violation and passes. A warn rule can never fail a build, which is what makes it safe to introduce a rule in warn first and watch what it would have caught.