Description

You will lead data engineering efforts using Azure Databricks and PySpark to build and maintain scalable ETL pipelines.

Responsibilities

• Design and implement complex ETL processes using Spark and Python.
• Conduct thorough design and code reviews to ensure technical quality.
• Perform requirement reviews and provide accurate work effort estimations.
• Manage offshore teams and oversee their code review processes.
• Document application development workflows and technical specifications.
• Apply unit and end-to-end testing best practices to all code deployments.

Required Skills

• 7+ years of experience in data engineering or related roles.
• Expertise in Azure Databricks and Spark.
• Strong proficiency in Python and PySpark.
• Deep understanding of ETL concepts and relational database technologies.
• Advanced SQL skills and experience with MS SQL Server.
• Experience working with XML and JSON file formats.
• Ability to work onsite in Branchville, NJ.
• Strong written and oral communication skills for stakeholder management.
• Degree in any graduate field.

Preferred Skills

• Experience with Azure Data Factory (ADF) platforms and frameworks.
• Familiarity with Agile methodologies and Jira.
• Knowledge of the Insurance domain.

Key Skills

Education

Any Graduate