Description

You will lead the design, development, and automation of data pipelines and quality frameworks to support data modernization and cloud migration efforts.

Responsibilities

• Build and automate data ingestion, transformation, and aggregation pipelines using Azure Data Factory, Databricks, Spark, Snowflake, and Kafka.
• Design and maintain metadata-driven pipelines to enable self-service data capabilities.
• Develop data quality, audit, and monitoring frameworks to ensure pipeline reliability and automated alerting.
• Lead the migration of data from on-prem platforms like IBM Netezza to cloud environments.
• Define engineering standards, design patterns, and CI/CD automation in collaboration with Enterprise Architecture.

Required Skills

• 5+ years of experience in Enterprise Data Management or Data Engineering.
• 3+ years of hands-on experience building metadata-driven pipelines using Azure Data Factory and Databricks/Spark.
• 5+ years of experience in data analysis and wrangling using Python, PySpark, or Jupyter Notebooks.
• Expert-level SQL knowledge across databases including Snowflake, Netezza, Oracle, SQL Server, MySQL, or Teradata.
• Experience with source control and build/release engineering using Azure DevOps or GitLab.
• Proficiency with SQL, Python, and Databricks.
• Experience working within Agile/Scrum and DevOps environments.
• Bachelor's degree in Computer Science.

Preferred Skills

• Experience with reporting tools such as PowerBI, Tableau, or OBIEE.
• Functional knowledge of Terraform, Azure CLI, PowerShell, or containerization with Kubernetes and Docker.

Key Skills

Education

Bachelor's Degree