Description

You will lead large-scale data migrations from IBM DataStage to Databricks on AWS, owning the technical execution and validation of these modernization projects.

This role is on-site.

Responsibilities

• Lead and manage end-to-end data migrations from legacy platforms like IBM DataStage to Databricks, ensuring seamless transition of complex ETL workflows.
• Translate visual DataStage parallel jobs and sequences into optimized Python/PySpark scripts or Databricks Notebooks, avoiding lift-and-shift anti-patterns.
• Implement CI/CD integration using Git and automated deployment processes for Databricks assets, leveraging Databricks Asset Bundles.
• Build automated reconciliation frameworks to compare row counts, checksums, and aggregate sums between legacy outputs and new Databricks targets.
• Orchestrate production-ready streaming and batch ETL pipelines using Delta Live Tables (DLT) and Databricks Workflows.

Required Skills

• 5+ years of experience in data engineering, with specific focus on migrating from IBM DataStage to Databricks on AWS.
• Advanced SQL skills for building modular analytics workflows, utilizing CTEs, and writing high-performance queries in Databricks SQL Analytics.
• Strong proficiency in Python and PySpark for building, optimizing, and debugging complex data transformation scripts and distributed DataFrames.
• Deep understanding of Apache Spark ecosystem, including cluster execution flow, memory allocation, and driver/worker node management.
• Experience with Delta Lake architecture, including ACID transactions, data skipping, partition strategies, and automated compaction.
• Hands-on experience with Delta Live Tables (DLT), Unity Catalog for governance and lineage, and Auto Loader for incremental ingestion.
• Familiarity with CI/CD tools, Git version control, and Databricks Asset Bundles (DABs) for automated deployments.
• Experience with cloud infrastructure on AWS, including object storage, IAM, and network security configurations.

Preferred Skills

• Experience with external orchestrators like Azure Data Factory or Airflow.
• Knowledge of AI coding assistants and agent tools for schema mapping and dependency analysis.

Key Skills

Education

Bachelor's degree