The pipeline failed at 4 a.m. I found out at 6, because I was the one on call that week.
Before my first cup of coffee, I was writing the email nobody wants to write… a message to all of leadership letting them know that reporting would be delayed, that we were working on it, and that I would follow up when I knew more. (There is a special kind of silence and desire to put your head in a toilet that follows that email.)
Then came the crime scene. A massive orchestration workflow, dozens of dependencies, and one failure buried somewhere in the middle of it. Hours of pulling logs and walking the chain backward until there it was, something we have all seen before… a schema change in files being dropped by a vendor we ingested from. Upstream. Uncommunicated. And here is the punchline. When we finally got the vendor on the phone, the schema change turned out to be an error on their end. Our pipeline did nothing wrong. It broke anyway.
The vendor re-dropped corrected files. And here is where the story turns from bad luck into a design lesson. The orchestration was not built to restart from the point of failure. No checkpoints, no idempotent stages, no way to resume mid-stream. The only option was to rerun the entire workflow from the beginning. This is the story that drives me to share what I’ve shared in other videos and articles, that is a design flaw, and I did not build it. I inherited it. But that is exactly the point. Almost every data engineer reading this is running orchestration they inherited, making payments on design decisions someone else signed off on years ago. The person on call pays for choices made long before their tenure. A full day, gone. Leadership had no reports. Worse, frontline salespeople lost all visibility into the customer information they used to close deals. And the strategic project I was supposed to be working on that week, a customer service platform migration, slipped by days. One bad file from one vendor, and the whole week reorganized itself around a fix.
Every data team I have worked with in 26 years carries a version of that morning. It is a debt we have all paid. Not technical debt. Something worse, because it never shows up on a roadmap and is rarely planned for.
I call it the Maintenance Mortgage. It is the recurring payment your team makes, every week, just to keep what you already built from falling over. Sometimes it looks like my 4 a.m. schema change. Sometimes it is late-arriving data, a silent quality failure poisoning a dashboard for three weeks, or a drifting model serving confident, wrong answers. The failure mode varies. The payment never stops. And in most shops, the mortgage was signed by someone who left years ago. You are just the one making payments.
You never pay down the principal. You only pay interest.
At the Data + AI Summit, Databricks announced Genie ZeroOps, a background AI agent aimed squarely at that mortgage. Before we get to what it is, we need to talk about how big the payment actually is. Because the number is worse than most leaders think.
The 53 Percent Problem
Two independent reports landed on the same figure this spring. Fivetran’s 2026 Data Connectivity Report and dbt Labs’ State of Analytics Engineering, a survey of 4,200 practitioners, both found that roughly 53 percent of enterprise data engineering time goes to maintaining existing pipelines rather than building anything new.
Read that again. More than half.
And it gets worse at scale. Fivetran found that organizations running more than 200 active pipelines spend an average of 61 percent of engineering time on maintenance. The more you build, the more of your team you convert into a repair crew.
Do the math on your own team. If you have ten data engineers at a fully loaded cost of $150K each, the Maintenance Mortgage is costing you roughly $800K a year in salary spent keeping the lights on. Not innovating. Not shipping the AI use cases your board keeps asking about. Fixing.
Here is the part that should genuinely concern leadership: AI is making this worse, not better. Coding assistants have made it faster than ever to ship pipelines and models. Every one of those assets becomes something that can break at 2 a.m. We accelerated the build side of the equation and left the operate side exactly where it was. The gap between how fast teams can create and how much they have to maintain is widening every quarter.
That gap is the actual problem ZeroOps is trying to close.
What ZeroOps Actually Is
Strip away the launch-keynote language and ZeroOps is a background agent that lives inside the Databricks platform and runs a four-step loop on your production assets: detect, assess, remediate, verify.
Detect. It continuously monitors pipelines, jobs, tables, and ML models. Not just hard failures. It watches data quality metrics for the silent stuff, the pipeline that runs green while quietly loading garbage.
Assess. When something breaks, it traces root cause through Unity Catalog lineage. The full dependency graph. So instead of an alert that says “job failed,” you get “this table broke because a team two hops upstream changed a schema on Tuesday.”
Remediate. It drafts a proposed fix.
Verify. This is the step that made me sit up. ZeroOps shallow-clones production data into an isolated sandbox (zero-copy, scoped permissions, network isolation) and tests the proposed fix against real data without touching production. For an ML model, it goes further: it trains a candidate on corrected features and evaluates it against the same eval suite the production model was held to. Not a generic benchmark. Your benchmark.
Then, and this matters, a human approves. Issues land in an inbox prioritized by severity, with the root cause analysis and proposed fix attached. Nothing hits production without sign-off. You configure which assets it watches and what it is allowed to do.
Now replay my 4 a.m. morning against that loop. Detect happens while I am still asleep. Assess replaces my hours of log spelunking with a lineage trace that points straight at the vendor’s files. The proposed remediation is waiting in an inbox when I sit down at 6:00, and I spend my morning reviewing a diagnosis instead of writing an apology to leadership. I have not used ZeroOps. Nobody outside the preview has. But I have lived the exact failure it was designed for, and the design maps onto that morning step for step. That is worth taking seriously. It is also worth verifying, which we will get to.
Why This Couldn’t Be a Coding Agent
The obvious question: can’t Cursor or Claude Code do this already?
No. And the reason is architectural, not a marketing claim.
Data operations failures are usually data problems, not code problems. A schema change upstream. Bad records propagating through a dependency chain. Corruption that sits in a table for weeks. A coding agent staring at your repository cannot see any of that, because the evidence isn’t in the code. It’s in the metrics, the logs, the run history, and the lineage graph. That telemetry lives inside the platform.
And there is a harder wall: production data is sensitive and governed. You cannot responsibly hand an external agent read access to your production tables so it can “investigate.” An agent that operates under Unity Catalog governance, inside the platform’s permission model, can do the verify step safely. An outside agent can’t.
The moat here isn’t the model. It’s the lineage and the governance. (Which, not coincidentally, is why Databricks spent years building Unity Catalog before building this.)
What This Actually Means for Your Team
Here is where I want to move past the product and talk about the humans.
If ZeroOps delivers even a fraction of what it promises, the job of the data engineer shifts from firefighter to fire marshal. From writing the fix at 4 a.m. to reviewing a proposed fix at 9 a.m. with a root cause analysis already attached. From spelunking through logs to exercising judgment.
That is not a smaller job. It is a more senior one.
Think about what your team could do with even half of that 53 percent back. Remember what my broken pipeline actually cost. Not just the day of troubleshooting. It delayed a platform migration, a strategic project with a business case behind it, because the person driving it (me) got pulled into forensics. That is how the Maintenance Mortgage really collects: it does not just eat hours, it eats momentum on everything else. The data contracts initiative that never gets funded with time. The semantic layer. The documentation nobody writes. The proactive quality checks that would prevent next quarter’s incidents. Maintenance crowds out exactly the investments that would reduce future maintenance. Teams stay trapped in the cycle not because they lack skill but because they lack hours. Breaking that loop is worth more than any single feature.
But I want to be honest about the other side, because the “review and accept” workflow carries its own risk. If your engineers approve fixes they don’t understand, you haven’t automated operations. You have automated the erosion of your team’s understanding of its own systems. The skill that becomes precious in a ZeroOps world is the ability to look at a proposed fix and know whether it is right. That skill comes from having fixed things yourself. Leaders will need to be deliberate about keeping that muscle alive, especially in junior engineers who may never get the reps their seniors got.
Judgment doesn’t scale unless you protect the way judgment gets built.
The Fine Print, Because There Is Always Fine Print
ZeroOps is entering private preview, starting with jobs, pipelines, tables, and ML workloads. Apps and Lakebase are on the roadmap. Which means most of you cannot touch it yet, and none of us has independent evidence it works at enterprise scale.
So here is what I would watch when it reaches your workspace. Not the demo. These numbers:
Root cause accuracy. How often is the diagnosis actually right?
The unedited-approval rate. What percentage of proposed fixes do your engineers approve without modifying? That number is the real trust signal. If it’s low, you bought a very expensive suggestion box.
Mean time to resolve, before and after. Measure your baseline now, while you still have one.
Cost per incident, net of agent compute. Consumption billing means the agent’s investigation runs are not free. Nobody at the keynote mentioned that part. I am mentioning it now.
The Verdict
The Maintenance Mortgage is real, it is measurable, and at 53 percent it is the single largest line item in how your data team spends its life. Any credible attempt to refinance it deserves your attention.
ZeroOps is a credible attempt. It is built on the right foundation (lineage, governance, sandboxed verification) and it keeps a human in the loop, which is exactly where the human should be. It is also a preview product from a vendor with every incentive to deepen your platform commitment, and it will live or die on trust metrics we cannot see yet.
And one caveat the marketing will not volunteer: an agent can propose a fix, but it cannot retrofit good architecture. The orchestration I inherited would have burned that day with or without ZeroOps, because a full rerun was the only path the design allowed. ZeroOps lowers your monthly payment. It does not rewrite the mortgage. Only your team can do that, and they need hours to do it. Which is, in the end, the strongest argument for buying those hours back.
So do not wait for the tool to save you. Start now: measure your team’s maintenance ratio this quarter. Fivetran pegs top-quartile organizations at 35 percent or below. Know your number before an agent promises to change it.
Because the strategy question was never “should we automate operations.” It was always “what will we do with the time.” Teams that answer the second question will win with or without ZeroOps.
Strategy > Syntax. Even when the syntax fixes itself.
If you are running Databricks and you do not know your maintenance ratio, or you know it and you do not like it, that is what a Databricks Lakehouse Readiness Review is for. We map the current state, separate the risks that matter from the noise, and leave your team with a roadmap they can execute. It starts with a strategy call, which costs you thirty minutes and nothing else.
And either way, reply with your maintenance ratio. I am collecting real numbers from real teams, and I will share what I learn in a future issue.