You are vibe coding your way into a career dead end.
You are right now sitting in front of Cursor or Claude Code. You’re dumping a raw repository prompt into an agent harness and generating a 500+ line PySpark pipeline, and then blindly hitting merge. You think you’re multiplying your productivity.
You think AI has transformed you into a high-leverage architect overnight, but you’re wrong. You’re just an expensive copy and paste operator blowing tokens on something that we used to do by copying and pasting Stack Overflow responses.
When that pipeline ends up stalling in production at 2 a.m. because an agent hallucinated column transformations that don’t exist in your DeltaLite schema, who do you think is going to get that on-call alert? Who is going to have to stand up in front of the VP of Finance and justify why your cloud compute bill doubled while their monthly finance reports still aren’t right?
Who is going to shoulder all of that blame? I’ll give you a hint, it’s not your AI.
The industry is rapidly filling up with engineers who blindly let their LLMs write code that they don’t understand, inside workspaces that they don’t govern, and using agent sessions that they never clean up. You end up trading your technical discipline for short-term speed.
You aren’t multiplying your enterprise value: you are multiplying your technical debt at $25 per million Opus 5 output tokens. (You know that is what you are using!)
The Reality About Ungoverned Agents
Now, I don’t blame you for taking the easy path. 26 years ago, when new IDEs and automated tools made bad logic fast, I fell into that same exact trap. Feels good to move fast. You get that hit of dopamine that you accomplished something, even if it was really the AI doing it. Gives you a temporary high just to see that screen scroll with hundreds of lines of working code in mere seconds.
However, speed without direction and guidance is just faster collision with production reality and loss of business trust.
Let’s look at a real engineering problem rather than some canned tutorial dataset: Ingesting raw NOAA weather telemetry into Databricks medallion architecture using PySpark, Delta Lake, and Unity Catalog. On paper, the blueprint is pretty straightforward.
Bronze: This is your receiving dock for all your raw JSON payloads containing nested weather station metrics, your quality flags, and all of your irregular timestamps.
Silver: The sorting floor where your schemas are enforced, your structure types are flattened, your null ratings are quarantined, and records are deduplicated based on station ID and observation time.
Gold: The storefront where your business roll-ups, your daily temperature extremes, and climate analytics are all modeled into star schema for high-concurrency executive reporting.
If you just point to an unguided AI agent at a blank repo and say, “Build me a NOAA weather ingestion pipeline,” it will one shot build it, and it will run.
It is also going to hard-code your storage credentials, mix complex aggregation logic into your bronze landing layer, drop your schema validation rules, and write unoptimized full table scans that are going to burn your Databricks DBUs.
And when you accept that pull request without understanding the underlying platform architecture, you haven’t solved the business problem. You just created an unauditable, unmaintainable piece of tech debt that is going to fail the exact moment your source data schema drifts!
Gen.ai makes writing syntax trivial, but syntax was never the bottleneck. Strategy is.
Hacking Meta-Harnesses With Scott Haines
Last week I sat down for a livestream on what I have dubbed Live Data Friday with my friend Scott Haines. Scott is a staff developer advocate at Databricks and co-author of Learning Spark as well as Delta Lake: The Definitive Guide, and we spent our session digging Databricks’ most recent open-source project called Omnigent.
Omnigent is a meta-harness where, instead of forcing you to work in isolated silos where Cursor knows one thing, clob knows another, and codex is completely blind to both, Omnigent links multiple agent harnesses together over a shared local workspace. It creates a single execution pane where your sub-agents can collaborate and debate across the same repository.
The promise of the meta harness is context orchestration. As Scott and I began driving Omnigent through real refactoring tasks, we ran head first into one of the core failures of modern AI workflows, and that’s context bloat.
When you have three sub-agents executing terminal commands, running unit tests, inspecting directories, and fixing syntax errors, they generate this massive operational just noise. Terminal output, stack traces, failed git commits, raw markdown files all get dumped directly into that session buffer.
Keep feeding that raw, unorganized session history back into the LM, and the model doesn’t get smarter, it gets confused. Recent research and live production tests prove that repeating instructions or cluttering an active context window actually lowers model confidence and causes hallucination rates to increase. When an agent has to process 150,000 tokens of raw, trial-and-error chatter just to write a 10-line SQL join, it loses track of your original constraints.
Raw context becomes absolutely a liability, and distilled memory becomes your asset… but how do you do that?
You can’t run enterprise platforms on agent context windows that look like your junk drawer. That’s why I started working on the dreaming skill, and Scott’s helped me on our stream to refine and demonstrate it.
But What Is The Dreaming Skill?
Your brain does not retain every single raw sensory detail from every day of your life. When you fall asleep at night, your brain carries out a critical distillation process. It discards the clutter, compresses all of your repeated experiences, consolidates core lessons, and puts it all into your long-term memory. You wake up the next morning with a clear mental status. This is also why sometimes you wake up at 3 am realizing how to solve a problem in your code from the previous day.
AI agents need that same operational discipline, that same process.
Anthropic touched on the concept with their research into agentic memory distillation, but I wanted to take that theory and apply it directly to data engineering inside of Omnigent, inside of this meta harness.
The dreaming skill takes that messy raw chatter of a long coding session, discards all the transient terminal logs and failed syntax attempts, and distills all that into core ground truth in a lightweight structured memory file, such as agents.md or, in this case, a project dream file.
Instead of forcing your sub-agents to read 100,000 tokens of historical trial and error every time it boots up, the model reads a 2 KB distilled snapshot of what your architecture rules are.
During our stream, Scott installed the skill locally, pointed it at our repository, and asked Claude to run a dream cycle. The agent scanned the existing codebase and identified that our project was tracking instruction sets across Claude.md and agents.md, and updated its internal search scripts to handle both file structures.
When the GitHub API through a transient 500 server error while Scott was trying to push the changes, the agent didn’t panic or dump garbage into the workspace. It identified the transient failure, retried the operations, opened a clean pull request, and provided a concise summary of architectural changes that it made.
That is the difference between an unguided prompt and a governed agent. GUIs are for operators, code is for engineers, and your distilled dream context is for the architects.
Strategy > Syntax: What models to use for which workloads?
One of the concepts that are built into Omnigent when you’re docked inside of Databricks is the intelligent router. That intelligent router analyzes the question and routes the context to the agent best suited to answer the question.
I know you’re all out there using Opus 5 right now, probably max, for everything that you’re doing, because why not? That’s what you want: the best agent for everything, right? Well, it’s not an efficient use of your tokens, and it’s not an effective way to work.
During our sessions, one of the concepts that Scott and I brought up was a simple real-world metaphor, and that’s the grocery store. Each week, you sit down at the start of the week to plan your family’s meals. You use your high reasoning, strategic part of your brain. You balance your budgets, look at everybody’s schedules, evaluate what you need to prepare for the week, and then you write a precise list of ingredients and things that you need at the store.
When you get to the grocery store, you don’t need that high-level strategic reasoning to pick up a carton of milk from the shelf. Just follow the list! When you walk into the store without a list and try to plan your entire week’s architecture while standing in the cereal aisle, you end up with a shopping cart full of junk food and half the ingredients missing for dinner.
That is exactly how most developers misuse AI models. They dump raw, unorganized prompts into high-tier reasoning models like Opus 5 or GPT-5.6 Sol and ask them to generate basic boilerplate SQL or write routine PySpark wrappers. Then they’re burning high-tier API budgets on low-ROI tasks. The right way to build data pipelines with AI agents is to split your planning from your execution.
The architect phase. This is where you use your high-tier models. You’re using a top-tier reasoning model like Opus 5 to evaluate your system trade-offs. Define your governance rules. Establish your medallion-schema boundaries, and you’re using a plan to do this. It writes all the ground truth specifications and your detailed plan or your grocery list into your Agents.md file.
Next is your execution phase. This is where you’d use your mid-tiered models. Think GPT-5.6 terra instead of sol. Think sonnet instead of Opus. The execution agent doesn’t need to guess the architecture decisions that it should make or understand what the partitioning strategy is, because that’s all baked into your plan. It reads that to kill pipe dream file that just got distilled from your plan by using the drain skill and executes that exact PySpark transformation required for your bronze-to-silver clean steps. (In Databricks, hopefully you are using declarative pipelines with expectations for your quality rules.)
When we applied the strategy to our NOAA weather repository, architecture fell into place naturally.
The strategic planner established that bronze_prod.NOAA must remain an immutable append-only ledger of all the raw JSON script strings with added ingestion metadata.
It specified that our silver_prod.weather must enforce type schemas for station IDs, convert raw observation timestamps to UTC, and quarantine invalid air temperature ratings.
It defines Gold_prod.climate_analytics as a dimensional model structured strictly for low-latency BI queries.
After all those rules were locked into the project’s distilled memory, the mid-tier execution subagents wrote the logic cleanly without hallucinating schemas or breaking UC naming permissions.
In my previous article, “The 10-Year Junior,” I wrote about experienced developers who get trapped in the comfort zone because they end up relying on vendor GUIs and tribal knowledge rather than mastering the core engineering principles. Today, the new comfort zone is unmonitored AI usage, relying on an LLM to hold all of your architecture without enforcing any type of distilled memory file. That’s simply the modern incarnation of tribal knowledge shields.
The Agent Memory Governance Protocol
Now, if you want to transition from an operator who blindly accepts LLM code diffs to an architect who governs AI execution plans, you need a repeatable operational framework. Here is the five-step protocol for managing agent memory in production data engineering.
Step 1: Separate Strategy from Execution
Never allow an execution agent to actually touch your repo without a pre-approved specification file (your /plan) Use your highest reasoning models strictly to plan your system boundaries, define table contracts, write structure, and all constraints into an explicit markdown file, then pass that specification to your lower tier models for the actual work.
Step 2: Execute Daily Memory Distillation (The Dream Pass)
At the end of every feature build or debugging session, invoke a dreaming pass. Instruct your agent harness to:
• scan your active workspace
• discard transient terminal outputs and failed syntax attempts
• compress new architectural decisions
• write the updated state back to your memory file
Then start a new context window pointing to that dream file.
This is what the dream skill we are working on is doing… (dreaming-skill repo)
Step 3: Lock down Workspace Security Boundaries
Now, before opening a shared agent session or hooking multiple harnesses into Omnigent, audit your workspace security. Verify that local tools and file search utilities explicitly ignore .env files, git credentials, and production access tokens. Never give an AI agent unsandboxed access to an environment that holds your credentials!
Step 4: Prosecute the Diff
Every pull request generated by an agent should be treated like it is code written by a junior on their first day. Never click merge without reviewing every single line of code!
Look specifically for:
• optimized joins
• dropped data quality checks
• missing transaction boundaries
• hard-coded literals
• for spark table writes… overwrite mode instead of append or merge
Only then should you actually merge back into main.
Step 5: Audit Token & Compute Efficiency
Evaluate all of your agent workflows through a strict FinOps lens. Make sure that you are understanding how many tokens your agents consume relative to the actual business value of the code that you’re producing. If an agent session burns 200,000 tokens to output a basic data transformation, your context is unorganized! Refactor your memory files until your agents can execute tasks in tight, high-confidence bursts, or write it yourself. Notice what every single step in this protocol has in common. Each step requires a human in the loop who understands how production platforms actually work and actually break. Managing AI agents isn’t about writing clever prompts or using the brand-new frontier models.
It’s about:
enforcing platform governance
ensuring idempotency
making sure that you’re not blowing up your cloud compute bill
and modular, maintainable code
Gambill Insight
AI will not replace data engineers. It is however the next evolution of your skillset. Just like engineers who blindly copy-pasted code from stack, those that don’t understand the output, are able to effectively guide it, and have a solid understanding of the foundations, will be left behind.
Stop using AI as a crutch to avoid critical thinking. Start using it as a disciplined force multiplier for platform architecture.
If your enterprise engineering team is spending thousands on AI tools and cloud compute without seeing reliable, production-grade outputs, stop guessing. [Book a Strategy Call with Gambill Data]. We audit architectures, implement governance, and build platforms that survive the boardroom.