Description

You will design and implement scalable data pipelines and end-to-end data warehouse projects within a Big Data environment.

Responsibilities

• Build scalable streaming solutions using Hadoop/Big Data stack (HDFS, Ozone, Hive, Impala, Kafka).
• Develop data pipelines using Git, Shell, and Spark (Java/Python) on Hadoop or Object storage (S3/CEPH).
• Execute incremental data loads and ETL processes across multiple projects.
• Debug production issues via root cause analysis and implement mitigation plans.
• Provide technical leadership to junior team members and perform POCs for complex problems.

Required Skills

• 8+ years of experience building scalable data pipelines.
• Expertise in Scala, Python, and PySpark.
• Hands-on experience with Apache Spark, Spark SQL, and Hadoop.
• Proficiency with Kafka and NiFi.
• Strong SQL knowledge and experience with Oracle and Netezza databases.
• Experience with ETL and incremental data loading.
• Ability to work within India time zones to coordinate with global teams.
• Strong analytical and debugging skills for production environments.

Preferred Skills

• Development experience in Python-based AI/ML workloads on Hadoop.
• Experience with APIs, Unix environments, and Agile methodologies.

Key Skills

Education

Any Graduate