Description

You will design and maintain PySpark applications to process large-scale datasets across batch and streaming environments.

This role is on-site.

Responsibilities

• Design and test PySpark applications for data from relational databases, NoSQL, batch files, and real-time streams.
• Implement efficient data transformations, aggregations, and error handling to ensure data integrity.
• Optimize Spark jobs through partitioning, caching, and configuration tuning.
• Build ETL processes to ingest and integrate data from data lakes and streaming platforms.
• Collaborate with data scientists and architects to translate requirements into technical solutions.

Required Skills

• 5+ years of hands-on big data development experience.
• Proficiency in PySpark and Apache Spark.
• Experience with ETL development and orchestration tools (Apache Airflow or Luigi).
• Strong knowledge of SQL and NoSQL databases.
• Experience working with data lakes and distributed datasets.
• Deep understanding of data processing principles and big data best practices.
• Bachelor's degree in Computer Science, Information Technology, or a related field.

Key Skills

Education

Bachelor's degree in Computer Science, Information Technology