Description

You will design, build, and maintain large-scale data processing pipelines.

Responsibilities

• Architect, construct, test, and maintain ELT pipelines using Hadoop and Spark SQL.
• Manage and optimize infrastructure to ensure data reliability and performance.
• Design and implement data models and schemas for enterprise data warehouses and data marts.
• Write and maintain Python scripts for various data applications.
• Monitor system performance, identify bottlenecks, and implement solutions for high data availability.

Required Skills

• 10+ years of professional experience in data engineering.
• High proficiency in the Hadoop ecosystem (HDFS, MapReduce, Yarn) and Apache Spark.
• Strong competency in Scala for large-scale data processing.
• Mastery of advanced SQL and deep experience with relational databases and data warehouses.
• Proficiency in Python for data manipulation and scripting.
• Hands-on experience with workflow management tools, specifically Apache Airflow.
• Familiarity with Unix shell scripting for automation and system management.

Key Skills

Education

Any Graduate