Description

You will design and implement scalable data pipelines and streaming solutions within a Big Data environment.

Responsibilities

• Develop end-to-end data warehouse projects and scalable streaming solutions using the Hadoop stack.
• Build data pipelines using Git, Shell, and Spark on Hadoop or Object storage like S3/CEPH.
• Debug production issues, perform root cause analysis, and implement mitigation plans.
• Provide technical leadership to junior team members and perform POCs to explore new technology solutions.
• Coordinate with international teams by working flexible hours to align with India time zones.

Required Skills

• 7+ years of experience building scalable data pipelines.
• Expertise in Scala, Python, and PySpark.
• Proficiency with Apache Spark, Spark SQL, and Hadoop.
• Experience with Kafka and NiFi for data orchestration and streaming.
• Strong SQL knowledge and experience with Oracle and Netzza databases.
• Hands-on experience with ETL and Incremental Data Load processes.
• Knowledge of HDFS, Ozone, Hive, or Impala.
• Ability to work within Unix environments and use Shell scripting.
• Any Graduate degree.

Preferred Skills

• Development experience in Python-based AI/ML workloads on Hadoop.
• Experience working in Agile teams and using APIs.

Key Skills

Education

Any Graduate