DEV Community

Cover image for How IFCO Cut a 7-Hour dbt Job to 2 Hours 20 Minutes on Databricks
AI Data Legends
AI Data Legends

Posted on Originally published at aidatalegends.com

How IFCO Cut a 7-Hour dbt Job to 2 Hours 20 Minutes on Databricks

A data pipeline does not need a radical rewrite to become dramatically faster. Sometimes, the biggest gains come from making an incremental model truly incremental—and finding out what the query engine actually executed rather than trusting what the transformation code appears to say.

That is the central lesson from IFCO's work with Databricks Forward Deployed Engineering. IFCO, whose reusable packaging network operates across more than 50 countries, processes large volumes of tracking data to calculate business-critical metrics such as cycle time, loss, breakage, washing costs, and pool size. Late-arriving events complicate the job: historical records may need reconciliation even after an earlier run has reported its results.

According to Databricks, IFCO reduced the runtime of its core semantic-layer job from approximately seven hours to 2 hours 20 minutes—a reduction of about 66%. Daily compute costs fell by 58%, and the team retired the nightly full refresh it had previously needed to maintain KPI accuracy. The consolidation model also scanned about 75% fewer rows, while only around 3-5% of the relevant assets needed to be recomputed on each run.

The changes were not one magic setting. They combined more deliberate incremental writes, execution-plan-based diagnosis, and a different way of operating the dbt dependency graph on Databricks.

For data engineers maintaining large transformation projects, the lesson is practical: optimize the rows and files a model touches, then make every model independently observable and recoverable.

Why it matters

At large scale, the cost of a transformation depends not only on the SQL it expresses but also on how the database reads and writes data to execute it.

An incremental dbt model is meant to avoid rebuilding a whole table when only a small part has changed. Yet a configuration that looks incremental can still trigger broad scans or writes. A changing timestamp, an unbounded time window, unnecessary columns, or a poor match between the model's grain and its write strategy can quietly turn a small update into a large operation.

This becomes particularly important when source events can arrive late. A pipeline needs to reconcile recent history without rescanning or rewriting far more data than correctness requires. Too narrow a window risks missing corrections; too broad a window wastes compute.

The operational problem grows when hundreds of models run inside a single dbt command. If one model fails near the end of a long job, diagnosing the failure and deciding what to rerun can be harder than fixing the SQL itself. Aggregate logs also make it difficult to see which model is responsible for a runtime or cost regression.

IFCO's approach addresses both dimensions: reduce unnecessary data work inside each model, and reduce the operational opacity of the overall graph.

1. Diagnose the query that actually ran

One of the first lessons from IFCO's optimization work is deceptively simple: compiled dbt SQL is not necessarily the complete story of what Databricks executes.

dbt compiles model code and macros into SQL, but the runtime execution path can add behaviour associated with materialization and incremental strategy. Reviewing the compiled statement alone may therefore fail to explain the scans and writes visible in production.

Databricks and IFCO used actual execution information from query history to investigate the problem. That allowed them to examine the executed query plan and look for evidence such as rows scanned, write volume, spill, and the operators responsible for expensive work.

The distinction is important. A model may look incremental in the repository but still touch too much of its target table. An optimization should be grounded in observable runtime behaviour rather than a guess based on the SQL's appearance.


This is an excerpt. The full article continues with:

  • Measure the result, not the configuration change
  • Break the dbt graph into individual Databricks tasks
  • Use AI-assisted diagnostics as an evidence-backed review process
  • What data and AI engineers should do
  • Limitations and open questions
  • Key takeaways

Read the full article on AI Data Legends → — with diagrams, code and sources.

Originally published on AI Data Legends. Each post here is an excerpt; the full articles live at aidatalegends.com.

Top comments (0)