Description

Design and manage scalable data processing pipelines and middleware services for large-scale data systems.

This role is hybrid.

Responsibilities

• Build and maintain data ingestion and processing solutions using Hadoop ecosystem tools (HDFS, MapReduce, Hive, Impala, PySpark) and Teradata.
• Develop and manage batch and real-time data processing workflows and pipelines.
• Create and query GraphQL-based queries and optimize SQL for datasets up to 30-30TB.
• Develop REST APIs and middleware services using Python.
• Automate job scheduling and process optimization using Autosys and Unix shell scripting.

Required Skills

• 5+ years of experience in Big Data engineering.
• Expertise in Hadoop ecosystem: HDFS, MapReduce, Hive, Impala, PySpark.
• Proficiency with Teradata.
• Strong Python skills for middleware services and REST API development.
• Experience with Autosys for job scheduling.
• Proficiency in Unix shell scripting.
• Bachelor’s or master’s degree in Computer Science, Engineering, or related field.

Preferred Skills

• Experience with Graph databases.
• Familiarity with Agile development practices and secure coding protocols.

Key Skills

Education

Any Graduate