Description

You will design, build, and maintain large-scale data pipelines using a big data stack.

Responsibilities

• Build and automate data pipelines using Hadoop, Hive, PySpark, and Python.
• Design data models and implement database schemas for efficient storage and retrieval.
• Manage job scheduling and orchestration using Autosys.
• Integrate data services with Amazon AWS S3 object storage and manage associated security.
• Troubleshoot pipeline failures and apply transformation logic to complex datasets.

Required Skills

• 5+ years of hands-on experience in data engineering.
• Proficiency with Hadoop, Hive, PySpark, and Python.
• Experience with Amazon AWS S3, including object storage and security.
• Strong understanding of database design and modeling (MySQL or standard relational databases).
• Hands-on experience with Autosys for job scheduling.
• Competency in Unix/shell scripting and CI/CD pipeline implementation.
• Experience with PowerBI and Dremio.
• Degree in any field (Any Graduate).

Preferred Skills

• Exposure to GCP cloud data engineering.
• Experience in financial services or relevant domain knowledge.

Key Skills

Education

Any Graduate