Description

You will design, develop, and maintain scalable data pipelines and cloud-based data solutions, focusing on migrating on-premise Hadoop sources to Azure.

This role is on-site.

Responsibilities

• Design and implement ETL workflows and secure data solutions for ingesting real-time claims data.
• Manage the full data engineering lifecycle, including research, proof of concepts, development, testing, and deployment.
• Build diagnostic tools to troubleshoot, maintain, and optimize petabyte-scale data solutions.
• Develop CI/CD workflows on Azure to support continuous deployment for data projects.
• Maintain technical documentation regarding data definitions, transformations, and processes to ensure governance.

Required Skills

• 5+ years of experience working with Azure cloud.
• Proficiency in Python or Scala.
• Hands-on experience with Azure Databricks and Azure Data Factory (ADF).
• Experience with Kafka for real-time data integration.
• Experience building UI via React or Angular.
• Knowledge of CI/CD workflows.
• Ability to migrate SSIS workflows into cloud environments.

Preferred Skills

• Experience with large-scale petabyte data processing.

Key Skills

Education

ANY GRADUATE