Years ago, when the industry began its mass migration from on-premises servers to the cloud, I worked alongside a highly capable data engineer. For a decade, he had been the resident master of on-prem SQL Server. He knew every quirk of DTS, and later SSIS, tuning local server performance down to the bare metal.
But when the strategic mandate came down to move our pipelines to the cloud, dug his heels in like a stubborn mini-donkey. He refused to learn Azure. He refused to touch cloud orchestration. He was convinced that the cloud was a trend, a loss of control, and that his deep, localized SQL Server expertise would always be inherently more valuable.
He didn’t lose his job immediately. He just finished out his career maintaining legacy databases while the rest of the industry, and the high salary that came with it, moved on without him. He chose to become an artifact of years past.
I am bringing this up because data engineering has reached another such inflection point. If you are a data engineer today and you are not actively integrating AI agents and LLMs into your daily workflows, you are that on-prem SQL Server engineer. You are already legacy.
I recently spoke with Joe Reis and he also wrote about it this week (link at the end of the article), but something has changed. If you haven’t actively used AI in 6 months because you thought it wasn’t good enough, you should know the inflection point came in the last 6 weeks.
AI is no longer a novelty or a speculative tool for the future. It is fundamentally reshaping the speed, scale, and economics of how we build data platforms right now. To demonstrate exactly what I mean, I want to break down two specific interactions I had with AI this week that fundamentally accelerated how Gambill Data architects solutions.
1. Architecting Governance: The OpenClaw ERD Application
At Gambill Data, our standard is absolute: we do not build raw pipelines; we build data assets the business can trust. A non-negotiable component of that trust is rigorous data modeling. To support both our enterprise consulting engagements and my executive coaching, I recently began architecting a custom Entity-Relationship Diagram (ERD) tool that I’m referring to as ERA (Entity-Relationship AI).
I deployed an AI agent via OpenClaw to handle the heavy lifting. The tech stack I chose was a Next.js frontend, a Python backend, and DuckDB as the foundational engine. (While DuckDB is highly effective for this phase and serves as a reliable fallback, the strategic roadmap is to migrate the backend to LakeBase as we prepare the product for potential market launch, alongside future integrations for Snowflake, Synapse, and on-prem instances).
I did not give the agent rigid, pre-defined constraints. Instead, I operated as the Principal Architect, treating the AI like a highly capable, senior-level developer. We worked iteratively. I provided feedback on the UI, directed the visualization standards, and outlined the security parameters.
The results we achieved in just 5 to 6 hours of my time were staggering. Here is what we successfully implemented:
Databricks Unity Catalog Integration: We built a direct integration that reads the information schema from Unity Catalog. This allows users, particularly my coaching clients using the Databricks free edition, to model against their actual data warehouse environments instantly.
Automated Standards Validation: I directed the agent to build both deterministic and probabilistic comparison engines. Users can now upload their organization’s standard governance documents, and the tool outputs a direct report detailing any schema violations against those specific standards.
Blast Radius Impact Analysis: We engineered a feature that calculates the deterministic and probabilistic “blast radius” of any schema change. If you alter a table or a field name, the tool immediately identifies every downstream table impacted. (The next iteration will extend this radius to downstream pipelines and Power BI reports).
Here is the reality check: I do not know Next.js. I know Python, DuckDB, LakeBase, and enterprise architecture. I stepped in to review and PR the backend logic, but the AI generated a highly functional, secure, and visually polished Next.js frontend in hours. Now, my brother is a front end dev, and he doesn’t know it but I’ll probably ask him to review the code soon. But, OpenClaw (connected to Codex) completely bypassed the syntax barrier, allowing me to focus 100% of my cognitive effort on integrating Unity Catalog and engineering the blast radius logic.
That is the modern engineering workflow. If you are still hand-coding frontend scaffolding instead of architecting the governance engine, you are wasting your client’s time.
2. Executing the Medallion Architecture: The Databricks Assistant
My second experience this week was a masterclass in how AI is transforming the granular mechanics of enterprise data platforms.
We champion Databricks as the definitive standard for unified analytics. However, the day-to-day reality of moving data from Bronze to Silver still requires tedious syntax. I was tasked with promoting a highly messy Salesforce dataset into our Silver layer.
Instead of writing the PySpark boilerplate manually, I pointed the native Databricks AI assistant at the Bronze data and gave it explicit architectural instructions. I ordered it to:
Eliminate duplicates.
Enforce strict data types.
Scrub a series of specific inconsistencies known to plague this particular Salesforce instance.
Standardize the schema by converting all legacy
PascalCasecolumn names to our enterprisesnake_casestandard.Execute a
dropnacommand on our critical business keys.
The AI generated the Databricks notebook, and it executed flawlessly, on the first run!
I do not remember the specific regex it used for the Pascal-to-snake-case conversion. And that is exactly the point. I shouldn’t have to memorize regex. My job is to ensure that the data landing in the Silver layer is structurally sound and financially accurate.
The assistant not only nailed the transformations and the data quality rules (like dropping null business keys and deduplication), but it also successfully generated a clean, highly effective MERGE statement to write the final output into the Silver Delta table.
By offloading the syntax generation to the Databricks assistant, I compressed hours of coding into minutes of validation. This is how we drive high-margin ROI. We focus on the pipeline’s strategic impact, not the commas in the Spark SQL.
The Industry Consensus: The Role is Elevating
I am not the only one seeing this writing on the wall. Earlier this week, I sat down with Joe Reis to record an upcoming episode of his podcast, and we spent a significant portion of our time unpacking this exact shift.
Joe recently published an excellent piece on the future of data engineering career paths in the age of AI, and our conversation heavily echoed the same reality: AI is not killing the data engineer; it is forcing the role to evolve. The bottom is falling out for junior-level syntax generation, but the ceiling is rising exponentially for engineers who understand systems, business value, and data trust.
If your primary value to an organization is writing boilerplate ETL scripts, your job is highly vulnerable. The data engineers who will command premium rates and lead the industry going forward are those who transition from “code writers” to “architectural directors.” You must become the “Human in the Loop”, directing and orchestrating the AI, validating its outputs against business requirements, and guaranteeing the reliability of the platform.
As promised here is the link to Joe’s article as well:
The Reckoning Is Already HereEarlier this week, I published a piece called 2028 — The Great Data Reckoning. It was supposed to be satire. A riff on the Citrini Research “2028 Global Intelligence Crisis“ paper that rattled the stock markets early this week, except pointed squarely at the data industry. A fictional memo from 2028 looking back at how everything unraveled. A faux maste…Read more6 months ago · 30 likes · 8 comments · Joe ReisJoin Me at DataTune Nashville: Stop Guessing, Start Adapting
Adaptability is no longer a soft skill; it is a critical technical requirement. Because this shift is so vital to our industry right now, I am dedicating my upcoming talk to this exact topic.
On Saturday, March 6th, at 1:00 PM, I will be speaking at the DataTune Conference at Belmont University in Nashville, Tennessee. My session is focused entirely on Continuous Learning in the Age of AI for Data Engineers.
In this session, we will bypass the theory and focus on pragmatic survival. I will share real-world stories from my 25 years in the field, including more on the engineers who adapted versus those who didn’t. We will break down exactly how to evaluate new technologies, when to deploy generative AI for pipeline automation, and when to enforce proven, rigid practices to protect the business.
If you want to understand how to build sustainable learning habits and transition your career from a legacy syntax-writer to a modern data architect, you need to be in the room.
Gambill Data is proud to offer our network a discount on registration.
Use the code Gambill26 for 20% off your ticket. Register here: https://datatuneconf.com/tickets.html
The transition to AI-assisted engineering has already arrived. Stop resisting the current. Leverage these agents to scale your output, elevate your architectural focus, and build data platforms the business can actually rely on.
I look forward to seeing you in Nashville.