Description

You will manage the full data engineering lifecycle from research and proof of concept to deployment and maintenance.

Responsibilities

• Design and implement scalable, secure data solutions and ETL workflows to migrate data from on-premises and Hadoop sources to Azure cloud.
• Develop and manage data pipelines at petabyte scale using Databricks and Azure Data Factory to migrate SSIS workflows.
• Build CI/CD workflows on Azure to ensure continuous deployment for all projects.
• Implement instrumentation and monitoring within pipelines to detect data quality issues and internal problems before they impact users.
• Develop diagnostic tools and processes to troubleshoot, maintain, and optimize production solutions.
• Maintain high-quality documentation for data definitions, transformations, and processes to support governance and security.

Required Skills

• 8+ years of active development experience with Python and .NET.
• 8+ years of experience with UI technologies, specifically React.
• Expertise in Terraform and Azure Infrastructure.
• Experience ingesting data from Hadoop sources to the cloud.
• Proficiency with Python, Hadoop, AWS, R, and Java.
• Bachelor's degree in Computer Science.

Key Skills

Education

Bachelor's degree in Computer Science