Description

Lead the design and optimization of large-scale data processing systems within Databricks and AWS environments.

Responsibilities

• Design, develop, and maintain data pipelines using Databricks and PySpark to process large-scale datasets.
• Optimize Apache Spark batch processing workflows and fine-tune existing jobs for performance and reliability.
• Build and maintain streaming data pipelines and troubleshoot bottlenecks within the data architecture.
• Implement data governance, security, and compliance best practices within Databricks environments.
• Collaborate with Data Scientists and Analysts to provide efficient access to relevant datasets.

Required Skills

• 5+ years of experience as a Data Engineer with a heavy emphasis on Databricks.
• Extensive hands-on experience building and optimizing data pipelines using PySpark.
• Mandatory fundamental knowledge of AWS tools.
• Proficiency in Spark SQL and the DataFrame API for dynamic data transformations.
• Strong knowledge of SQL, data modeling, and ETL processes.
• Experience using Python or Scala for advanced filtering logic in Databricks notebooks or scripts.
• Solid understanding of Databricks components including clusters, notebooks, jobs, and libraries.
• Understanding of distributed systems principles and their application in message brokering.
• Proven expertise in optimizing systems for low-latency and high-throughput performance.

Preferred Skills

• Bachelor's degree in Computer Science, Engineering, or a related field.

Key Skills

Education

Bachelor's degree