Description

You will design and maintain large-scale data pipelines on Databricks and AWS. You own the end-to-end ETL lifecycle, from ingestion to optimization.

Responsibilities

• Develop and maintain data pipelines using Databricks and Spark/PySpark.
• Implement Delta Lake solutions to ensure data quality and performance.
• Integrate data from multiple sources into an S3-based data lake.
• Optimize pipeline performance and troubleshoot complex data issues.
• Collaborate with architects and business analysts on solution design.

Required Skills

• 9+ years of experience in data engineering or related roles.
• Strong hands-on expertise with Apache Spark and PySpark.
• Proficiency in SQL and Python for data manipulation and scripting.
• Deep experience with the Databricks platform.
• Strong understanding of ETL processes and data pipeline architecture.
• Experience with AWS services, specifically S3, EMR, or Glue.
• Familiarity with orchestration tools such as Airflow.
• Healthcare data domain experience is required.
• Bachelor's degree in Computer Science or a related field.

Key Skills

Education

Bachelor's degree