Description

Design and build ETL pipelines across on-prem and cloud environments. Own the data engineering lifecycle from ingestion to delivery.

Responsibilities

• Build ETL pipelines using Sqoop, Hive, MapReduce, and Spark.
• Develop complex data transformations and in-memory computations using Python and Scala.
• Design logical, physical, and dimensional data models using Erwin.
• Orchestrate workflows and jobs using Airflow and Automic.
• Monitor cluster performance, set up alerts, and provide production support.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency in Spark, Python, and Scala.
• Experience with Sqoop and Hive.
• Strong SQL skills and experience with BigQuery.
• Hands-on with Git and Jenkins for CI/CD.
• Experience writing UNIX/Linux scripts for complex file formats.
• Knowledge of Kafka producer and consumer implementation.
• Ability to perform system administrator tasks and troubleshoot running applications.
• Degree in any graduate field.

Key Skills

Education

Any Graduate