Description

You will design and build scalable data pipeline platforms and automated frameworks for data ingestion, transformation, and consumption.

Responsibilities

• Design and implement automated Spark-based frameworks to facilitate data ingestion and consumption.
• Build high-performing data pipeline platforms using Hadoop, Apache Spark, MongoDB, Kafka, and object storage.
• Implement security protocols including Kerberos authentication, encryption of data at rest, and role-based access control via Apache Ranger.
• Develop automated testing frameworks to perform data validation.
• Collaborate with infrastructure engineers, architects, and data analysts to design big-data infrastructure and scalable solutions.

Required Skills

• 7+ years of experience in big data engineering.
• Proficiency in Scala, Java, and Python.
• Experience building and enhancing Scala frameworks and Spark.
• Hands-on experience with MongoDB and Kafka.
• Knowledge of Hadoop ecosystems and object storage architecture.
• Experience implementing Kerberos and Apache Ranger.
• Bachelor's degree in Computer Science.

Key Skills

Education

Bachelor's degree in Computer Science