Description

Design, develop, and maintain scalable ETL/ELT pipelines using Databricks and PySpark.

This role is on-site.

Responsibilities

• Build batch and real-time data processing solutions using Apache Spark.
• Implement data transformation workflows using Python, PySpark, and Spark SQL.
• Develop Delta Lake solutions and optimize data processing performance.
• Manage Databricks notebooks, jobs, and workflows for production releases.
• Troubleshoot pipeline issues and ensure data quality and logging.

Required Skills

• 5+ years of experience with Databricks and Python programming.
• Expertise in Apache Spark, PySpark, and Spark SQL.
• Strong SQL skills and experience with ETL/ELT pipeline design.
• Hands-on experience with Delta Lake and lakehouse concepts.
• Knowledge of cloud platforms: AWS, Azure, or GCP.
• Familiarity with Git, CI/CD, and Agile practices.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with Airflow, Azure Data Factory, or Databricks Workflows.
• Knowledge of Kafka or streaming frameworks.
• Understanding of data governance and security best practices.

Key Skills

Education

Bachelor's degree