Description

Key Responsibilities:

• Design and implement distributed data processing solutions using Apache Spark, Hadoop, Flink

• Develop and maintain Spark applications for data transformation, aggregation, and ETL processes using Scala, Java, or Python

• Utilize Apache Ozone for storing large-scale datasets, ensuring efficient data access and management in a distributed environment

• Manage and optimize HDFS and Apache Ozone, Kafka for scalable and fault-tolerant storage.

• Develop ETL pipelines for batch and real-time data ingestion and transformation.

• Implement and ensure data validation, data security, integrity, and compliance across big data platforms.

• Monitor and troubleshoot performance issues in large-scale clusters.

• Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions.

• Automate workflows and improve operational efficiency using scripting and orchestration tools.

Required Skills & Qualifications:

• Strong expertise in Apache Spark (Core, SQL, Streaming).

• Hands-on experience with Hadoop ecosystem (HDFS, YARN, MapReduce).

• Proficiency in Apache Ozone for object storage and integration with Hadoop.

• Solid programming skills in Java , Scala , or Python.

• Experience with Hive, HBase , and Kafka is a plus.

• Knowledge of cluster management and resource optimization.

• Familiarity with Linux/Unix environments and shell scripting.

• Understanding of data security, governance, and compliance standards.

• Experience with cloud-based big data platforms

• Exposure to containerization (Docker, Kubernetes) for big data workloads.

• Knowledge of CI/CD pipelines for data engineering projects

Key Skills

Education

Any Gradute