Description

You will build and manage scalable data pipelines and data processing workflows.

Responsibilities

• Design, build, and maintain scalable data pipelines using Hadoop, Hive, PySpark, and Python.
• Integrate data with Amazon S3 and manage object storage security and connectivity.
• Perform data modeling and database design using MySQL or equivalent systems.
• Manage job scheduling with Autosys and automate workflows via Unix/Shell scripting and CI/CD pipelines.
• Troubleshoot performance issues and optimize data processing tasks.

Required Skills

• 4+ years of hands-on experience with Hadoop, Hive, PySpark, and Python.
• Proficiency with Amazon AWS S3.
• Experience with Autosys job scheduling.
• Strong data modeling and database design skills.
• Experience with Power BI and Dremio for data visualization.
• Proficiency in Unix/shell scripting.
• Experience with CI/CD tools and pipeline integration.
• Ability to transform, cleanse, and validate data using business logic.
• Any Graduate degree.

Preferred Skills

• Exposure to Google Cloud Platform (GCP) and cloud data engineering concepts.
• Prior experience in the financial services domain.

Key Skills

Education

Any Gradute