Description

Design and optimize scalable data processing systems and pipelines using PySpark, Hadoop, and Cloudera.

This role is on-site.

Responsibilities

• Develop and maintain large-scale ETL processes and data pipelines using PySpark and the Hadoop ecosystem.
• Build, manage, and optimize workflow orchestration and scheduling with Apache Airflow.
• Implement data quality checks, validation logic, and governance standards across Cloudera platforms.
• Collaborate with data scientists and analysts to translate requirements into robust data solutions.
• Monitor pipeline performance, troubleshoot issues, and ensure adherence to security and compliance protocols.

Required Skills

• 5-9 years of hands-on experience in big data engineering or data architecture roles.
• Strong proficiency in PySpark and Apache Spark for data transformation and processing.
• Deep knowledge of the Hadoop ecosystem and Cloudera platform management.
• Experience designing and optimizing data pipelines for scalability and reliability.
• Proficiency in Python and SQL for data querying and framework development.
• Practical experience with Apache Airflow for workflow orchestration.
• Understanding of data governance, security best practices, and quality assurance.
• Strong analytical and problem-solving capabilities with clear communication skills.

Preferred Skills

• Experience optimizing data workflows for high-performance and low-latency requirements.
• Familiarity with emerging big data technologies and continuous architecture improvement.

Key Skills

Education

Any Graduate