Description

Role description

Role purpose. Take an approved design to a deployed and validated bundle, operating the multi-agent system and owning the quality of everything that goes into dev. This person does not write all the code; they are accountable for it.

Candidate profile summary

Data engineer with sound independent technical judgment, comfortable reviewing and validating code they did not write, debugging in production environments, and knowing when to ask rather than assume. Will work with AI-assisted code generation tools as a normal part of the workflow.

Required experience

• Data engineering or ETL development.
• Python and SQL in production environments.
• Hands-on experience with Apache Spark
• Databricks (nice to have: Asset Bundles).
• Experience with version control and Pull Request workflows — code review as a regular practice, not an occasional one.
• Nice to have: exposure to CI/CD in Azure DevOps.

Key Responsibilities

• Agent Operation and Auditing: Interact with the multi-agent system, put the plan.md under scrutiny, and thoroughly review the generated code bundle (detecting bottlenecks such as data skew, excessive shuffles, or incorrect data typing).
• Parity Testing and Reconciliation: Design and run data parity validations certifying that the Databricks output matches the legacy source load 100% (volume, values, and schemas).
• Exception Management and Escalation: Independently resolve minor code warnings and promptly escalate NEEDS-HUMAN events, pattern changes, or semantic inconsistencies to the Architect.
• Code Quality and Security: Remediate findings from SAST tools (e.g., Fluid Attacks), dependency vulnerabilities, and non-conformities before opening the PR.
• Delivery Cycle (Git Flow): Manage feat/TICKET-slug branches, ensure strict compliance with local pre-checks, and lead the resolution of REJECTED tickets or defects raised in QA/UAT.
• Transition Stability: Sustain operations, rapid anomaly remediation, and pipeline telemetry during the post-deployment hypercare window until formal handover to support.

Skills

Technical skills

Python · PySpark · SQL · T-SQL · Delta Lake · Spark · Git · Pull Requests · code review · CI/CD · Azure DevOps · pytest · YAML · ETL · SSIS · SQL Server Agent · Databricks (nice to have)

Nice to have: Basic reading of Alteryx/SSIS, pytest, code security, advanced Git

Key Skills

Education

Any Graduate