Description

You design and implement distributed data processing solutions for large-scale datasets.

This role is on-site.

Responsibilities

• Develop and maintain Spark applications for data transformation, aggregation, and ETL processes.
• Manage and optimize HDFS, Apache Ozone, and Kafka for scalable and fault-tolerant storage.
• Build ETL pipelines for batch and real-time data ingestion and transformation.
• Ensure data validation, security, integrity, and compliance across big data platforms.
• Automate workflows and improve operational efficiency using scripting and orchestration tools.

Required Skills

• 5+ years of experience in data engineering.
• Strong expertise in Apache Spark (Core, SQL, Streaming).
• Hands-on experience with the Hadoop ecosystem (HDFS, YARN, MapReduce).
• Proficiency in Apache Ozone for object storage and Hadoop integration.
• Solid programming skills in Java, Scala, or Python.
• Experience with containerization using Docker and Kubernetes.
• Familiarity with Linux/Unix environments and shell scripting.
• Knowledge of CI/CD pipelines for data engineering projects.

Preferred Skills

• Experience with Hive and HBase.
• Experience with cloud-based big data platforms.

Key Skills

Education

Any Gradute