# 5 Enterprise AI Prompts for Data Engineers

Copy-ready prompts for using AI on production-minded data engineering work. Each prompt narrows the model's solution space, bans common failure modes, asks for evidence before implementation, and keeps the engineer responsible for verification.

## Use these prompts safely

Never paste unsanitized customer data, credentials, PII or PHI, or proprietary schema metadata into public AI models. Sanitize inputs, strip identifiers, and use enterprise-approved environments for sensitive workloads. Treat every response as a first pass to review and validate, not as an architectural decision.

## Prompt 1: Legacy SSIS (.dtsx) to Vectorized PySpark

Convert this sanitized SSIS .dtsx XML into vectorized PySpark DataFrame transformations.

Hard Constraints:

- BANNED: collect(), toPandas(), row iteration (for row in ...), RDD maps, and Python UDFs unless mathematically required.
- Preserve all sources, joins, lookups, derived columns, casts, and error-redirect paths.
- Do NOT invent columns, tables, joins, filters, or fallback defaults.
- Output a Component Mapping Table BEFORE generating any code.
- Mark ambiguous SSIS components as [NEEDS_CONFIRMATION].

Required Output:

1. Component Mapping Table (SSIS Task -> Vectorized PySpark Equivalent)
2. Modular PySpark Functions (Pure transformations taking & returning DataFrames)
3. Schema Assumptions & Inferred Types
4. Production Validation Checklist

---

## Prompt 2: Adversarial Synthetic JSON Fixtures (Medallion Quality Gates)

Generate adversarial PyTest JSON fixtures to validate this data contract and pipeline ingestion logic.

Must Explicitly Cover:

- Null keys and missing nested objects
- Unannounced schema drift (unexpected extra attributes)
- Malformed timestamps (ISO-8601 vs Epoch milliseconds vs UTC offsets)
- Numbers formatted as string literals with whitespace
- Duplicate business keys across batches
- Empty arrays vs null arrays vs arrays with null elements
- Out-of-range categorical values

Hard Constraints:

- Use deterministic, reproducible mock values (no non-deterministic randomness).
- Zero real PII/PHI.
- Classify each test fixture case as: [ACCEPT], [NORMALIZE], [QUARANTINE], or [REJECT].
- Explicitly separate schema-level validation errors from business-rule rejections.

Required Output:

1. Fixture Inventory Matrix (Case Name, Failure Mode Tested, Expected Pipeline Action)
2. Raw JSON Test Fixtures
3. PyTest Test Suite (asserting Bronze-to-Silver quality gate routing)

---

## Prompt 3: Spark EXPLAIN Plan Diagnosis & FinOps

Analyze this Spark EXPLAIN EXTENDED physical execution plan.

Examine Specifically:

- Data skew indicators (uneven partition distribution)
- Missing partition pruning and Dynamic Partition Pruning (DPP) failures
- Costly shuffle exchanges and SortMergeJoin vs BroadcastHashJoin opportunities
- Predicate pushdown failures at the FileScan layer
- Redundant table scans and subquery re-evaluations

Hard Constraints:

- Base all findings strictly on physical plan evidence and supplied partition metrics.
- Clearly separate Confirmed Root Causes from Hypotheses.
- Do NOT recommend caching, repartitioning, or increasing cluster sizing unless mathematically justified by the plan.
- Provide step-by-step verification instructions in the Spark UI.

Output Format:

| Finding | Plan Evidence | Performance & DBU Impact | Recommended Fix | Spark UI Verification Step |

---

## Prompt 4: Stakeholder Email to ADR & JSON Schema Data Contract

Convert this stakeholder requirement email into:

1. An Architecture Decision Record (ADR)
2. A formal JSON Schema Draft 2020-12 Data Contract
3. An Unresolved Ambiguity Register

Hard Constraints:

- Do NOT invent requirements or assume business rules not explicitly stated.
- Explicitly separate Confirmed Requirements from Technical Assumptions.
- Flag any unstated edge cases or ambiguous definitions as [OPEN_QUESTION].
- Data Contract must specify: required fields, types, formats, regex patterns, nullability, and additionalProperties: false.
- Flag any PII, compliance, or governance considerations.
- Provide one valid payload example and three invalid payload examples violating the contract.

Required Output:

1. Architecture Decision Record (ADR) (Title, Status, Context, Decision, Consequences)
2. JSON Schema Draft 2020-12 Specification + Sample Payloads (1 Valid, 3 Invalid)
3. Unresolved Ambiguity Register (Open questions for stakeholder clarification)

---

## Prompt 5: Delta Lake DDL Schema Compatibility Review

Review this Delta Lake schema migration for backward and forward compatibility.

Required Output:

1. Migration Classification: [COMPATIBLE], [RISKY], or [BREAKING] with clear justification.
2. Production-Grade Delta Lake Migration SQL / PySpark DDL.
3. Rollback & Disaster Recovery Script.
4. Downstream Consumer Blast Radius Analysis (BI, dbt, Spark Structured Streaming).
5. Pre- and Post-Migration Validation Queries.

Hard Constraints:

- Classify column drops, renames, type narrowing, nullability tightening, and partition key changes as BREAKING.
- Do NOT assume downstream consumers tolerate unannounced column additions.
- Zero destructive DDL (no table rewrites or drops unless explicitly instructed).
- Preserve table properties (e.g., delta.columnMapping.mode, delta.minReaderVersion), partition layouts, and constraints.

---

Use the prompts to accelerate analysis and draft work, then validate the result in a safe environment with the people accountable for the data and production system.

Gambill Data | https://gambilldata.com/resources/