For the last five years, we convinced ourselves that the “Modern Data Stack” meant gluing together five different SaaS tools to move a CSV file from Point A to Point B. We write data in Spark, then spin up a separate container, install 400 Python dependencies, and run a heavy orchestration framework just to ask: “Is the customer_id null?”
This is insanity.
Every time you move data out of your compute layer just to validate it, you are bleeding cash. You are introducing latency. And you are adding a point of failure that has nothing to do with the data itself.
In 2026, the era of “resume-driven development” is over. We are moving toward the Lean Data Stack. And that starts with killing external testing frameworks in favor of native, compute-level validation.
Enter Databricks DQX.
Shift Left, or Die Trying
The core philosophy of the Lean Stack is simple: Strategy > Syntax .
If you are using dbt tests or Great Expectations, you are likely testing “after the fact.” The data has landed. The compute is spent. You are auditing the crime scene after the murder has happened.
DQX (Data Quality Expectations) is a Databricks Labs library that runs inside your PySpark compute graph. It shifts quality left. It is not an external policeman; it is the data’s judiciary system.
Why does this matter to you?
Cost: No egress fees. No separate orchestration containers.
Speed: Validation happens in-memory during the transformation.
Simplicity: One environment. No context switching.
The “AI Upgrade”: Stop Writing YAML
I generally hate “AI coding assistants” because they teach juniors to be lazy. But the 2025/2026 update to DQX with databricks-labs-dqx[llm] is different. It solves a specific ROI problem: Regex is a waste of billable hours.
Right now you are spending hours writing 50-line YAML files to define expectations. In the new world, you simply tell the engine what the business needs.
The Code:
# What You Are Doing: Manual Rule Definition (Slow, Brittle)
# rules = [DqxRule(name="valid_email", expr="email rlike '^[a-zA-Z0-9]+...'")]
# The New Way: Intent-Based Generation
from databricks.labs.dqx import suggestions
text_rule = "Ensure emails are valid, transactions are not in the future, and SKU is never null."
# Generates executable PySpark syntax instantly
rules = suggestions.from_string(text_rule)The Gambill Take: This isn’t magic. It’s efficiency. The AI handles the syntax (the commodity). You handle the logic (the value). If you are still billing your client for the time it takes you to Google a Regex pattern for a UK phone number, you are robbing them.
The Gambill Audit: Why dbt is Dead Weight in 2026
I’m going to say the quiet part out loud: If you are on Databricks, dbt is a redundant dependency.
For years, we treated dbt as the “standard” for transformation. But in 2026, we need to look at the ROI. If you are paying for Databricks, which gives you a world-class SQL engine, native Python support, and orchestration via Workflows. Then why are you adding a separate layer just to compile SQL?
Here is the Gambill Review of why I’m ripping dbt out of the Lean Stack entirely:
1. The Redundancy Tax
The dbt Way: You write SQL in a text editor. You wrap it in Jinja. You run a CLI tool to compile it. You send it to Databricks to execute.
The Lean Way: You write the SQL (or PySpark) directly in a Databricks Notebook. You schedule it with Workflows.
The Verdict: dbt is just a middleman here. It adds a build step, a compilation step, and a maintenance step. In a “Code to Cash” world, middlemen get cut.
2. The “Read-Write” Penalty (The Latency Tax)
The dbt Way: dbt is designed to run against data at rest. Your pipeline writes the table. Then dbt spins up. It reads the data back out to test it. This is “After-the-Fact” quality. You are paying for I/O twice.
The DQX Way: Validation happens in-memory within the compute graph. The data is checked before it ever hits the disk. We don’t pay for the round-trip. We don’t pay for the latency.
3. The Context Switch
The dbt Way: You force your engineers to context-switch between Python (for ingestion) and Jinja/SQL (for transformation). You maintain two repositories, two CI/CD pipelines, and two mental models.
The Lean Way: Unification. Ingestion, Transformation, and Quality (DQX) all happen in the same environment. A Senior Engineer knows that every context switch costs cognitive load and every new tool costs maintenance hours.
Here is the nail in the coffin…
In 2020 you were worried about freshness and referential integrity, but now we have DQX and UC.
Freshness: Databricks Lakehouse Monitoring tracks freshness automatically without a manual YAML test.
Integrity: Unity Catalog enforces Primary Key/Foreign Key constraints (informational or enforced) at the engine level.
Stop building “Resume Stacks” with five logos. Build Production Stacks that work. If Databricks can do the transformation and the testing natively, then dbt isn’t a tool, it’s just bloat.
The Architecture: “Fail Fast” is for Amateurs
Now that we've cleaned up the stack, let's clean up the philosophy.
The most dangerous advice given to Junior Engineers is “Fail Fast.” When my coaching clients are learning, I tell them to “Fail Loudly, Fail Often” because this is where you learn.
But, In a production pipeline processing 100 million rows of sales data, if one row has a bad timestamp, “failing fast” means the entire pipeline crashes. The dashboard goes blank. The COO or Sales VP calls me. And then I call you.
Production-grade engineering requires Anti-Fragility. We don’t fail; we Quarantine.
I covered the concept of Quarantine Tables extensively last year, and DQX has made the implementation trivial with the apply_checks_and_split pattern.
The Code:
from databricks.labs import dqx
# Define your policy
policy = dqx.Policy(
rules=rules,
error_strategy="quarantine" # The key differentiator
)
# Atomic Split
valid_df, quarantine_df = policy.apply_checks_and_split(source_df)
# The "Happy Path" continues uninterrupted
valid_df.write.mode("append").saveAsTable("silver_sales")
# The "Evidence" is stored for analysis and validation with the business
quarantine_df.write.mode("append").saveAsTable("quarantine_sales")Why this saves your job:
Business Continuity: The 99.9% of good data flows to the Silver layer immediately. The business keeps running.
Audit Trail: The
quarantine_dfdoesn’t just store the bad row; it tags it with the specific rule it violated.Recovery: You can fix the root cause in the source system on Monday morning, rather than debugging a crashed pipeline on Sunday night.
The Feedback Loop: Stop Being the Garbage Collector
There is a dirty secret in data engineering: most of you are not engineers; you are high-paid garbage collectors.
When you see bad data, like a country code listed as “UK” instead of “GB”, your instinct is to write a transformation rule to fix it. You write: CASE WHEN country = 'UK' THEN 'GB'.
Stop doing that.
Every time you write code to patch upstream negligence, you own that debt forever. You are telling the business, “It’s okay to enter garbage, because I will clean it up.”
The Quarantine Table is not a trash can. It is a Mirror.
The true power of the quarantine_df isn’t just isolating bad rows; it is using those rows to force the source system to improve. This is the ultimate “Shift Left”, shifting the responsibility back to the people who created the data.
The “Freeform Field” Nightmare Salesforce Admins can fix a dropdown menu. But they can’t fix a salesperson who is lazy. We all have that one field, usually “Notes” or “Deal Description”, where salespeople paste entire email threads, Zoom links, or write “TBD” to bypass a requirement.
Technical validation (Regex) can catch this, but the fix isn’t technical. It’s behavioral.
The Workflow:
Isolate: DQX flags the row where the “Phone Number” field contains text like “Extension 404” or “Ask for Bob.”
Attribution: You join that quarantine record back to the
sales_rep_id.The “Coaching” Report: You don’t send this to IT. You send a report to the Regional Sales Manager.
Headline: “Missed Commission Opportunities Due to Bad Data.”
Content: “Agent Smith lost 4 leads this week because the phone numbers were invalid.”
The Result: You are no longer nagging them about “schema compliance.” You are showing them how much money they are losing . When a Sales Manager sees that bad data entry is affecting their team’s quota, the behavior changes overnight. That is how you solve the “Freeform Field” problem without writing a single line of cleaning code.
The Verdict: Simplicity Scales
Hiring managers are tired of seeing “Tool Salad” on resumes. I don’t care that you know Airflow, dbt, Great Expectations, Soda, and Monte Carlo. I care that you can build a system that is cheap to run and impossible to kill.
The Lean Data Stack is about removing dependencies, not adding them.
If you are on Databricks, DQX removes the need for an external quality layer. It keeps the logic where the data lives. It reduces the “Orchestration Tax” to zero.
Next Steps:
Stop writing “Happy Path” tutorials.
%pip install databricks-labs-dqxin your next POC.Break it. Feed it garbage data. Watch it quarantine.
If you want to see this in action, I’m releasing a video walkthrough later this week where I deliberately try to break a DQX pipeline. Subscribe to the channel so you don’t miss the notification.
Event Note
I’ll be hosting a Databricks Workshop at DataTune in Nashville on March 6th. We’ll be doing a deep dive into building data assets that scale. If you're in the area, I’d love to see you in the workshop and afterwards and talk shop!