Description

Own the design, build, and optimization of scalable data pipelines and ETL workflows within the Hadoop ecosystem. You will ensure data quality, performance, and reliability across large-scale datasets.

Responsibilities

• Build and maintain data pipelines using Apache Spark and PySpark for efficient ETL processing.
• Develop, debug, and optimize complex SQL queries and Hive UDFs for data analysis.
• Ingest data from diverse sources including files, databases, and streaming platforms like Kafka.
• Manage Hadoop ecosystem components such as YARN and Zookeeper for cluster operations.
• Automate operational tasks using Unix scripting and handle data transfers via Sqoop.

Required Skills

• 5+ years of experience with Big Data technologies, specifically Hadoop, Hive, and Spark.
• Strong proficiency in PySpark for building scalable data pipelines and ETL workflows.
• Expertise in writing and optimizing complex SQL queries for large datasets.
• Deep understanding of Hive, including query development and Hive UDF creation.
• Experience with Unix scripting for automation and operational support.
• Knowledge of data ingestion tools like Sqoop and ecosystem components such as YARN and Zookeeper.
• Familiarity with Kafka for streaming data processing.
• Experience working with NoSQL databases and PL/SQL where applicable.

Preferred Skills

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Strong problem-solving skills with a focus on debugging and optimizing pipeline performance.

Key Skills

Education

Any Graduate