Description

You will manage large-scale financial data pipelines and real-time processing workflows within a Hadoop ecosystem.

Responsibilities

• Build and manage real-time data ingestion pipelines using Kafka to ensure scalability and fault tolerance.
• Develop and optimize complex Hive and Spark queries for efficient data retrieval and manipulation.
• Orchestrate complex ETL workflows and job scheduling using Oozie.
• Perform cluster management, troubleshooting, and performance tuning using Cloudera distribution and Ambari.
• Maintain data governance, security, and compliance standards for large-volume banking datasets.

Required Skills

• 10+ years of professional experience in data engineering.
• 8+ years of hands-on experience with the Hadoop ecosystem.
• Expertise in PySpark for data processing and transformation.
• Proficiency with Spark SQL and Apache Spark distributions, including Spark on YARN.
• Experience with Kafka for real-time data streaming and ingestion.
• Advanced knowledge of Hive for data warehousing.
• Solid command of Linux for file handling, process management, and system troubleshooting.
• Experience with YARN for resource management.
• Background in managing ETL processes and large-scale financial data.

Preferred Skills

• Familiarity with Hadoop administration tasks, including cluster monitoring and tuning.

Key Skills

Education

Any Graduate