During a call this week with one of my coaching students we were discussing the architecture review of a provider MDM pipeline. We were spot checking golden records generated across a provider database. One record immediately stood out.
The provider’s legal name was William. The legal name field in his golden record said “Illium”.
This was not a malicious change, database corruption, or fuzzy matching libraries going off the rails. It was the mathematical result of one naive survivorship rule.
most_frequent_name_wins
A high-value source had repeatedly reported the provider’s name as Ilium. That incorrect value appeared 14 times, while the correct value only appeared twice.
The rule he created for the frequency worked exactly as designed. It chose Ilium. Bad data beat the truth 14 to 2.
When data teams (and most consultants) pitch MDM, they show a clean diagram with several source systems feeding one polished box labeled “golden record”. The promise? A unified, trusted view of a real-world entity assembled from fragmented operational data.
That diagram rarely shows how that record earned the word trusted.
Deterministic algorithms can select absolutely trash data with complete consistency when its survivorship rules are built on bad assumptions. The dangerous failures are not always the pipelines that crash. Sometimes they are the pipelines that produce the wrong answer quietly, confidently, and on schedule.
Matching and Survivorship Are Different Decisions
Before we’re looking at failure, let’s separate two decisions that MDM implementations often merge together.
Entity matching asks: Do these source records describe the same real-world provider?
Field Survivorship asks: once those records belong to the same entity, which value should become the legal name? What about the address, phone number, specialty, or other mastered attributes?
Those two are not the same problem.
The provider records in this case had already been grouped into one entity cluster. In a production MDM pipeline, that decision should be based on multiple identifying attributes. The name should be one signal, not the only reason records belong together. A legitimate match can contain conflicting field values. In fact, resolving those conflicts is the reason survivorship exists to begin with.
Whether a cluster is formed through a deterministic anchor or a composite match, the critical failure here happened after the clustering occurred.
The Survivorship Engine evaluated the Legal Name values inside the Cluster:
Illiam: 14 occurrences
William: 2 occurrences
then it applied the rule for the most_frequent_name_wins, with character count used as the tiebreaker.
But there was no tie. The wrong value won by a landslide!
The problem wasn’t simply that one value appeared more often. The problem was treating each occurrence as independent evidence. Those 14 records were not 14 separate authorities confirming the provider’s identity. They were one high-volume process repeating the same mistake over and over and over again. That’s like allowing one witness to cast 14 ballots because they repeated the same statement 14 times!
Frequency measures repetition. It does not establish authority.
The Anti-Pattern: One Survivorship Rule for Every Field
The deep architecture mistake was applying one global survivorship strategy across fields that behave very differently.
Legal names and identity attributes
legal names do change, but not with the same frequency or operational meaning as contact information. They’re also vulnerable to abbreviations, transcription errors, stale source values, and inconsistent identity standards.
For Identity Fields, survivorship should normally prioritize:
an authoritative source;
validated identifiers;
verification status;
effective dates for legitimate changes;
steward review when trusted sources disagree.
Raw frequency can be supporting evidence. It shouldn’t be the final authority.
Physical addresses
addresses change more frequently and may represent different contexts. Think:
home address
mailing
billing
service
shipping
Ten old billing records should not automatically defeat one recently verified address.
Address survivorship usually needs some combination of:
address type
verification or deliverability status
most recently confirmed date
source system authority
effective start and end dates
Email addresses and phone numbers
The longest value is not necessarily best. The most frequent value may be stale. Contact channels should generally be evaluated using verification, consent, deliverability, and recency.
Regulated or public identifiers
In provider MDM, a validated NPI, or National Provider Identifier, can be a strong deterministic anchor. Records sharing a verified NPI will normally resolve to the same provider unless a conflict or exception rule is triggered.
That last piece matters. Even strong identifiers can be mistyped, misassigned, copied into the wrong source record, or presented alongside contradictory evidence. “Deterministic” should mean explainable and controlled, not unable to be challenged.
When one blanket rule is applied across an entire table, silent corruption isn’t an edge case. It’s an expected outcome waiting for the right pattern.
Why the Obvious Code Fixes Were Wrong
when the issue surfaced, the immediate instinct was to change the pipeline:
Lower the fuzzy match threshold
Change the distance algorithm
Add a hard-coded override for William without the W
None of those addressed the actual failure.
Changing a global matching threshold to separate one pair of names can break valid matches across millions of other issues. It also confuses matching with survivorship. Those records belong to the same provider cluster. The wrong field value was selected afterwards. Hard coding and if-else correction in PySpark would be worse. That turns a governed data decision into invisible application logic and starts that familiar march toward an unmaintainable swamp of edge-case policies. The pipeline needs to have executive survivorship policies. It shouldn’t become the place where every disputed business fact is permanently buried in code.
The immediate correction required two intentional actions:
Correct the source. A data steward traces the incorrect value to its operational origin and corrects the provider name there.
We run with lineage intact. Once the source contains the correct value, the entity resolution pipeline reprocesses the cluster and produces the correct golden record.
Sometimes the immediate fix isn’t a code fix. It’s a people and process fix, but that is only an incident response.
The durable, long-term fix is architectural:
Prevent repeated transactional records from maintaining false authority.
Make survivorship rules field-specific.
Provide a governed way to resolve conflicts that can’t be corrected at the source.
If the source system can’t be changed, use a stewarded override table with a reason, an owner, effective dates, and an audit history. Do not smuggle the override into a notebook and hope that the next engineer finds it and understands why it exists! Automated pipelines are execution engines and not truth arbiters.
A Defensible MDM Architecture
An MDM architecture that you can defend needs more than fuzzy matching and a final group buy. It needs:
explicit identity anchors
confidence bands
field-level survivorship
reversible lineage
operational ownership
Start with deterministic anchors.
Identify the strongest available keys for the domain before applying fuzzy logic. For healthcare providers, a validated NPI or tax ID may provide strong deterministic evidence. For customers or businesses without a universal identifier, like a D&B number, a composite of trusted attributes may be needed.
Deterministic anchors should accelerate clear matches, not eliminate exception handling. Conflicting identifiers, invalid formats, and suspicious reuse still need explicit controls.
Use fuzzy matching as evidence, not a verdict.
Fuzzy matching is valuable when names, addresses, and other descriptive fields contain legitimate variations, but a similarity score is evidence about whether records may belong together, not proof that one field value is true.
Use composite matching across multiple attributes and define confidence bands:
High confidence: merge automatically.
Gray: route to stewardship review.
Low confidence: keep your records separate.
The gray zone isn’t a pipeline failure. It’s the honest admission that some identity decisions require human interaction.
Configure survivorship by field.
Each master attribute should have an explicit decision policy based on its behavior and risk.
Here are a few examples:
Legal name: trusted source, verification status, and effective date.
Address: address type, verification, and recency.
Email and phone: verification, consent, deliverability, and recent confirmation.
Provider specialty: authoritative registry or credentialing source.
Public identifier: format validation, source authority, uniqueness, and conflict rules.
Store these rules as governed confirmation, configuration, or practical. Don’t scatter them across notebooks, stored procedures, and undocumented conditional logic.
I like to keep these rules in explicit tables with history tracking so that I know when a rule was created, by whom, and when it was deactivated.
Preserve source identity in a cross-reference bridge.
Never overwrite original source keys or irreversibly collapse records into the master table. Every generated master ID should map back to its contributing source records through a dedicated cross-reference bridge table that looks something like this.
master_id
source_system
source_record_id
effective_start_date
effective_end_date
is_active
match_method
match_confidence
decision_owner
This bridge table provides the one lineage needed to explain the golden record, reverse an incorrect merge, and determine which records contributed to each decision. In addition, it the ability to provide a customer 360 view across your data platform.
Use SCD2 history for relationship changes instead of relying on daily full table snapshots as your only audit mechanism. Full snapshots may have a place for recovery or regulatory retention, but they are a blunt and expensive substitute for explicit entity history.
Make stewardship part of the operational model.
Someone has to own ambiguous decisions.
You will need to define:
which confidence band requires review;
who can approve a merge or split;
how a source correction is requested;
when an override is allowed;
how the rationale and effective dates are stored;
how downstream systems receive corrected results.
If stewardship exists only as a rule that says, “email the data team, they’ll fix it,” it doesn’t exist.
The Larger Lesson
Here’s a short history.
2002: Teams built bad golden records with stored procedures.
2015: They built them with map-reduce jobs.
Today: They build them with modern, like-house platforms, expensive MDM products, and an LLM wrapper on top.
The tools evolve, but the failure mode doesn’t. Syntax does not fix strategy.
If your architecture relies on statistical heuristics without source authority, lineage stewardship, and accountability, you’re not creating a trusted master record. You’re actually building an automated system for amplifying whatever errors appear most often.
The goal isn’t to eliminate people. The goal is to make those decisions explicit, governed, auditable, and reversible.
Weekly Mission: Audit One Golden Record
Here’s a new section I’m adding to my articles: a weekly mission. This week I want you to pick one mastered entity and answer these five questions, or one entity that you wish was mastered.
Why did each field win? Can you explain the specific survivorship rule for the legal name, address, email, phone, and identifiers?
Are repeated rows manufacturing confidence? Does one high-volume source get more votes simply because it emits more transactions?
Can you trace the full lineage? Can the Golden Record be mapped back to every contributing source-system key through a 1 bridge?
Where is the grey zone? Which confidence range triggers steward review instead of an automatic merge?
How is a bad decision corrected? Is there a documented path for source correction, governed override, reprocessing, and, when necessary, splitting an incorrect cluster?
If you can’t answer those questions, the word “golden” is doing more work than the architecture actually is. Make the data decision that you can defend six months from now.
If your MDM platform can produce a confident wrong answer, the problem is rarely another line of PySpark. It’s usually the decision design around authority, survivorship, lineage, and stewardship.
Gambill Data provides independent data strategy and architecture reviews for organizations making consequential platform and governance decisions. If you want a senior, defensible read on your current design, book a strategy and architecture fit call.
If you’re an individual data professional working to build this level of architectural judgment, learn about our coaching program.
Related decision support
Data strategy consulting
Clarify ownership, governance, and the operating decisions behind trusted master data.
Review the service