Description

You will design and maintain PySpark applications to process large-scale datasets.

Responsibilities

• Design and test PySpark applications to process data from relational databases, NoSQL databases, batch files, and real-time streams.
• Implement efficient data transformations, aggregations, and error handling mechanisms to ensure data integrity.
• Optimize Spark jobs through partitioning, caching, and configuration tuning to improve performance.
• Build ETL processes to ingest and integrate data from data lakes and various streaming platforms.
• Collaborate with data scientists and architects to translate requirements into technical data solutions.

Required Skills

• 5+ years of hands-on experience in big data development.
• Proficiency in PySpark and Apache Spark.
• Experience with ETL development and data pipeline orchestration tools like Apache Airflow or Luigi.
• Strong knowledge of SQL databases and NoSQL databases.
• Experience working with data lakes and distributed datasets.
• Deep understanding of data processing principles and big data best practices.
• Bachelor's degree in Computer Science, Information Technology, or a related field.

Key Skills

Education

Bachelor's degree in Computer Science, Information Technology