Description

You will build and maintain production data pipelines using Databricks, PySpark, and Delta Lake. Your work ensures data reliability across batch and streaming workflows.

Responsibilities

• Develop and optimize ETL processes using Python and SQL within the Databricks environment.
• Implement Bronze/Silver/Gold medallion architecture for structured data management.
• Manage schema evolution and data quality monitoring to ensure pipeline integrity.
• Integrate with Kafka and orchestrate workflows using Airflow or Prefect.
• Handle metadata management and governance requirements for regulated environments.

Required Skills

• 8+ years of ETL experience, with at least 5 years specifically in Databricks.
• Strong proficiency in PySpark, Python, and SQL for data manipulation.
• Deep understanding of Delta Lake features and performance tuning.
• Experience with batch and streaming data processing architectures.
• Familiarity with data orchestration tools such as Airflow or Prefect.
• Knowledge of data quality frameworks and monitoring strategies.
• Ability to design scalable data solutions for healthcare or regulated industries.

Key Skills

Education

Any Graduate