Description

You design, develop, and deploy scalable ETL/ELT data pipelines using Apache Spark, PySpark, and Databricks. You optimize SQL queries for data transformation and analysis, collaborating with product owners and data architects to build data models, Delta Lake structures, and workflows. You ensure data quality, performance, and reliability while building job orchestration and monitoring solutions. You also develop and maintain CI/CD pipelines for Databricks notebooks and jobs on cloud-based platforms like Azure.

Responsibilities

• Design and deploy scalable ETL/ELT pipelines using Apache Spark, PySpark, and Databricks.
• Develop and optimize SQL queries for data transformation and analysis.
• Collaborate with product owners, data architects, and analysts to build data models and Delta Lake structures.
• Build job orchestration and monitoring solutions to ensure data quality and performance.
• Develop and maintain CI/CD pipelines for Databricks notebooks, jobs, and workflows.

Required Skills

• 10+ years of overall experience in data engineering or related fields.
• 3–5 years of hands-on experience with Databricks and Spark.
• Strong proficiency in SQL and data analysis techniques.
• Experience with ETL processes, data modeling, and performance tuning.
• Familiarity with Python or Scala for data engineering tasks.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with Azure cloud-based data platforms.
• Excellent problem-solving and communication skills.

Key Skills

Education

Bachelor's degree