Description

You will lead the architecture and development of scalable data processing systems and ETL workflows.

Responsibilities

• Architect and develop scalable data processing systems, data lakes, and ETL pipelines using PySpark.
• Fine-tune Spark applications through configuration tuning, memory management, and resource allocation.
• Manage and optimize Spark clusters for high availability using Kubernetes, YARN, or Mesos.
• Design efficient data storage and retrieval strategies using HDFS, S3, Parquet, and ORC.
• Collaborate with data scientists and engineers to translate requirements into distributed computing solutions.

Required Skills

• 10+ years of experience as a Lead Spark Developer or Data Engineer.
• Extensive hands-on experience with PySpark and Python.
• Deep understanding of distributed computing principles and architectures.
• Proficiency with Spark SQL, DataFrames, and Dataset APIs.
• Experience managing clusters with Kubernetes, YARN, or Mesos.
• Practical knowledge of HDFS, S3, Parquet, and ORC.
• Expertise in designing fault-tolerant and scalable data processing systems.
• Strong ability to tune Spark performance and resource management.

Key Skills

Education

Any Graduate