Description

You will design, build, and maintain scalable ETL/ELT pipelines and data processing solutions on Databricks. You own the full lifecycle of data workflows, from architecture to production support.

This role is on-site.

Responsibilities

• Develop batch and real-time data processing solutions using Apache Spark and PySpark.
• Implement Delta Lake architectures and optimize data processing performance through partitioning and caching.
• Create and manage Databricks notebooks, jobs, and workflows with rigorous data quality and exception handling.
• Support CI/CD pipelines, deployment automation, and production releases.
• Troubleshoot complex pipeline issues and collaborate with cross-functional teams to deliver requirements.

Required Skills

• Strong hands-on experience with Databricks and Apache Spark ecosystem.
• Proficiency in Python programming and PySpark for data transformation.
• Expertise in Spark SQL and complex SQL data transformation skills.
• Experience with Delta Lake and lakehouse concepts.
• Knowledge of ETL/ELT design and data pipeline architecture.
• Familiarity with cloud platforms: AWS, Azure, or GCP.
• Experience with Git, CI/CD, and Agile practices.

Preferred Skills

• Experience with Airflow, Azure Data Factory, or Databricks Workflows.
• Knowledge of Kafka or streaming frameworks.

Key Skills

Education

Bachelor's degree