Description

You will design and maintain scalable data pipelines and lakehouse architectures using Databricks and Spark.

This role is on-site.

Responsibilities

• Build and optimize ETL/ELT workflows to ingest and transform structured and unstructured data.
• Develop data lakehouse architectures using Delta Lake for efficient analytics.
• Optimize Spark performance and cluster configurations for cost-efficient processing.
• Implement real-time and batch processing pipelines using Kafka or Spark Streaming.
• Integrate Databricks with AWS, Azure, or Google Cloud Platform.

Required Skills

• 14+ years of experience in data engineering.
• Expertise in the Databricks platform.
• Strong proficiency in PySpark, Spark, and Spark SQL.
• Hands-on experience with Delta Lake architecture.
• Experience with AWS, Azure, or Google Cloud Platform.
• Knowledge of Kafka or Spark Streaming.
• Experience implementing CI/CD pipelines.

Preferred Skills

• Bachelor's degree.

Key Skills

Education

Bachelor's degree