Description

Design and build scalable data ingestion, processing, and analysis solutions for large-scale datasets.

This role is hybrid.

Responsibilities

• Develop and maintain data processing pipelines using Hadoop ecosystem tools (HDFS, MapReduce, Hive, Impala, PySpark) and Teradata.
• Build and manage middleware services and REST APIs using Python.
• Write optimized SQL queries for datasets up to 30-30TB and create GraphQL-based queries.
• Automate processes and job scheduling using Unix shell scripting and Autosys.
• Collaborate on performance optimization, secure coding practices, and Agile testing concepts.

Required Skills

• 5+ years of experience in Big Data engineering or related roles.
• Expertise in Hadoop ecosystem: HDFS, MapReduce, Hive, Impala, PySpark.
• Strong proficiency with Teradata.
• Advanced Python development for middleware and REST API integration.
• Experience with Unix shell scripting and Autosys job scheduling.
• Ability to write optimized SQL for large-scale data processing.
• Knowledge of Graph databases.

Preferred Skills

• Bachelor’s or master’s degree in Computer Science, Engineering, or a related field.
• Experience with real-time data processing workflows.

Key Skills

Education

Any Graduate