Description

You will develop and manage large-scale data processing pipelines within the Hadoop ecosystem.

Responsibilities

• Build and manage real-time data processing pipelines for critical financial applications.
• Develop and optimize complex Hive and Spark queries for efficient data retrieval.
• Orchestrate complex ETL workflows and job scheduling using Oozie.
• Manage Cloudera distributions, including cluster management, performance tuning, and troubleshooting.
• Implement data ingestion using Kafka to ensure scalability and fault tolerance.

Required Skills

• 10+ years of experience in data engineering or related roles.
• Expertise in PySpark for large-scale data processing and transformation.
• Deep knowledge of the Hadoop ecosystem and Apache Spark distributions.
• Proficiency in Hive, Spark SQL, and YARN.
• Experience with Cloudera distribution and Ambari for cluster monitoring.
• Hands-on experience with Kafka for real-time data ingestion.
• Strong Linux command line skills for file handling and system troubleshooting.
• Experience managing large volumes of financial data with a focus on governance and security.
• Experience with Oozie for ETL workflow orchestration.

Key Skills

Education

Any Graduate