Description

Design and implement scalable data-driven communication systems and ETL pipelines using Databricks on AWS or Azure.

Responsibilities

• Develop and optimize ETL pipelines from diverse data sources using Databricks.
• Implement standardized pipelines with automated testing, Airflow scheduling, and Azure DevOps for CI/CD.
• Manage infrastructure as code using Terraform and monitor system health with Splunk.
• Execute real-time data processing using Spark Structured Streaming and integrate outputs with REST APIs.
• Lead data engineering projects within Scrum and Agile frameworks to coordinate global delivery teams.

Required Skills

• 8+ years of experience developing and implementing ETL pipelines using Databricks on cloud platforms.
• Proficiency in Python and SQL.
• Hands-on experience with the Databricks platform and Delta Lake storage.
• Expertise in Spark, including PySpark and Spark SQL.
• Experience with Infrastructure as Code (AWS, Azure, or GCP).
• Proven experience working with insurance domain data.
• Strong ability to manage client discussions and technical escalations.
• Bachelor's degree in Computer Science.

Preferred Skills

• Experience with Airflow, Splunk, Kubernetes, Power BI, Git, or Azure DevOps.

Key Skills

Education

Bachelor's degree in Computer Science