Description

Lead the design and optimization of large-scale data processing systems within Databricks and AWS environments.

Responsibilities

• Design and maintain data pipelines using Databricks and PySpark for large-scale datasets.
• Optimize Apache Spark batch processing workflows for performance and reliability.
• Build and maintain streaming data pipelines, troubleshooting architectural bottlenecks.
• Implement data governance, security, and compliance best practices.
• Collaborate with Data Scientists and Analysts to provide efficient data access.

Required Skills

• 5+ years of experience as a Data Engineer with heavy emphasis on Databricks.
• Extensive hands-on experience building and optimizing pipelines using PySpark.
• Mandatory fundamental knowledge of AWS tools.
• Proficiency in Spark SQL and DataFrame API for dynamic data transformations.
• Strong knowledge of SQL, data modeling, and ETL processes.
• Experience using Python or Scala for advanced filtering logic in Databricks notebooks or scripts.
• Solid understanding of Databricks components: clusters, notebooks, jobs, and libraries.
• Understanding of distributed systems principles and message brokering.
• Proven expertise in optimizing systems for low-latency and high-throughput performance.

Preferred Skills

• Bachelor's degree in Computer Science, Engineering, or a related field.

Key Skills

Education

Bachelor's degree