Regulated enterprise analytics team · worked example
Escaping the 500-report migration trap
How a legacy reporting estate should be triaged, sequenced, and piloted before a line-by-line lift and shift to Databricks turns fifteen years of technical debt into metered cloud spend.
This is a worked example drawn from an observed situation and a published field note, not a delivered Gambill Data engagement. The organization is not named and no client outcome is claimed. It shows how the migration would be assessed and sequenced.
Client context
The situation.
A regional analytics director in a regulated enterprise was told the legacy environment was being deprecated and everything had to move to Databricks within twelve months. Roughly 500 automated reports ran from Windows Task Scheduler on a single VM. Four analysts kept them alive, two knew basic SQL, none knew PySpark, and the original authors had retired five years earlier. A systems integrator proposed a seven-figure, line-by-line translation of every script into notebooks.
Symptoms and risks
- About 500 scheduled reports with no consumption inventory
- Business rules living only in scripts written by people who had left
- A team that could not yet support the target platform
- A lift-and-shift proposal that would convert free local execution into metered DBU spend
Review approach
How the problem was examined.
The work focused on evidence and decision quality before prescribing implementation.
Tested where a 1:1 rewrite breaks: hidden inefficiency becomes visible spending
Traced report duplication into pipeline duplication
Checked upstream Unity Catalog readiness against downstream rewrite plans
Assessed how tribal knowledge would survive an outsourced translation
Engagement timeline
How the work unfolded.
Phases are listed in the order they happened. Durations appear only where they can be stated without exposing client detail.
- 01
Portfolio audit
Cross-reference every scheduled report against actual consumption. Decommission zombie reports, consolidate filter variants that query the same entity into one governed Gold table, and flag CSV-to-Excel export workflows for direct BI integration.
Migration scope defined by business capability rather than file count, before anyone writes code.
- 02
Upstream alignment
Map the consolidated list against the enterprise Unity Catalog roadmap into three queues: ready now, blocked on ingestion, and deprecation candidates. Freeze development on anything whose sources have not landed and passed validation.
- 03
Reference-architecture pilot
Build 10 to 15 representative reports to the standard the rest will follow: Bronze, Silver, and Gold separation, explicit business rules validated with analysts, completeness and freshness checks, defined scheduling and recovery, measured runtime and cost per workload.
A documented reference architecture and real evidence for estimating the remainder.
- 04
Team enablement
Pair external specialists with the internal analysts on the reference pipelines, using the organization's own data as the curriculum, so the handoff includes demonstrated capability rather than documentation alone.
Findings and recommendations
A sequenced path, not an unbounded backlog.
- The decision test: archive any report whose owner cannot name a decision it drove in the last 90 days
- The consolidation matrix: collapse filter variants of the same entity into one governed table
- The upstream contract gate: never write downstream code against schemas that are not live and validated
- The FinOps sanity check: project continuous DBU cost against shared serverless SQL before committing
- The day-two operational test: confirm the internal team can troubleshoot, modify, and monitor the result
Outcome
What changed.
Applied to the inventory above, this sequence produces a smaller, governed reporting estate the business trusts and the internal team can operate, with the migration counted in preserved business rules and retired work rather than rewritten notebooks.
Related service
Explore the Databricks lakehouse risk review
Review the service boundaries, typical deliverables, pricing context, and evidence before deciding whether to schedule.