Description

You will build and maintain big data processing pipelines using the Hadoop ecosystem. You own the full end-to-end software development life cycle (SDLC) for large-scale data solutions.

Responsibilities

• Develop Spark programs for large-scale data processing and analysis.
• Design and implement ETL workflows using Python and PySpark.
• Manage data structures and queries within Hive and Impala.
• Evaluate HDFS file formats to determine optimal storage solutions.

Required Skills

• 5+ years of experience in big data environments.
• Expertise in Python and Spark programming.
• Hands-on experience with PySpark and Hive.
• Proficiency in SQL and big data analysis.
• Practical knowledge of HDFS file formats and their trade-offs.
• Experience managing the complete SDLC.
• Bachelor's degree or equivalent graduate qualification.

Preferred Skills

• Background in ETL processes.
• Experience with PL/SQL.

Key Skills

Education

Any Gradute