Description

You will design, develop, and maintain scalable ETL pipelines on the Cloudera Data Platform.

Responsibilities

• Build and optimize ETL pipelines using PySpark to ensure data integrity and accuracy.
• Implement data ingestion processes from relational databases, APIs, and file systems into the data lake or warehouse.
• Transform and cleanse large datasets using PySpark to meet analytical and business requirements.
• Tune PySpark code and Cloudera components to optimize resource utilization and reduce runtime.
• Automate data workflows using Apache Oozie, Airflow, or similar orchestration tools.
• Monitor pipeline performance, troubleshoot issues, and maintain data quality through validation routines.

Required Skills

• 5+ years of experience in data engineering roles.
• Advanced proficiency in PySpark, including RDDs, DataFrames, and optimization techniques.
• Hands-on experience with Cloudera Data Platform (CDP) components such as Hive, Impala, HDFS, and HBase.
• Strong SQL skills and experience with Hive or Impala for data warehousing tasks.
• Experience with big data technologies including Hadoop and Kafka.
• Proficiency with orchestration frameworks like Apache Oozie or Airflow.
• Strong Linux scripting skills for automation.
• Any Graduate degree.

Key Skills

Education

Any Graduate