Description

Key Responsibilities

• Design, develop, maintain, and support scalable data pipelines using Big Data and AWS technologies
• Lead and support data platform migration initiatives (On-Prem to AWS Cloud)
• Develop and manage ETL/ELT processes using tools like SSIS, Pentaho, or similar
• Implement and manage AWS services such as EMR, S3, EC2, Redshift, Glue, and Airflow
• Build and optimize data workflows and orchestration pipelines using Airflow
• Work with real-time streaming technologies such as Kafka and Spark Streaming
• Perform data ingestion, transformation, and validation from multiple data sources
• Optimize SQL queries for performance and scalability
• Monitor system performance, troubleshoot issues, and ensure system reliability
• Collaborate with cross-functional teams including developers, architects, and infrastructure teams

Required Technical Skills

Cloud Technologies

• Strong experience in AWS Cloud services:
• EMR, EC2, S3, VPC, RDS, Redshift
• AWS Glue, IAM, CloudWatch, CloudFormation
• Airflow (or AWS Managed Workflows)

Databases

• Experience working with:
• SQL-based systems: SQL Server and PostgreSQL
• Data warehouses: Teradata, Redshift, Netezza

ETL Tools

• Hands-on experience with:
• SSIS

Programming & Scripting

• Strong proficiency in:
• SQL
• Shell scripting
• Good to have:
• Python

Operating Systems

• Strong experience in Unix/Linux environments.

Key Qualifications

• 10+ years of experience in Data Engineering / Big Data / Platform Engineering
• Hands-on experience in AWS and Hadoop ecosystems
• Experience in data migration projects (e.g., Hadoop to AWS, DB migrations)
• Strong understanding of data warehousing concepts and ETL design patterns
• Expertise in query optimization and performance tuning
• Experience supporting production data platforms
• Bachelor’s degree in Computer Science, Engineering, or related field.

Preferred Qualifications

• Experience in Banking / Financial Services domain
• Experience with data lake architecture and governance
• Knowledge of DevOps tools and CI/CD pipelines
• Experience in platform engineering and framework development.

Nice to Have

• Experience with tools like Ansible, Git, JIRA
• Exposure to NoSQL databases (HBase, MongoDB)
• Experience in data masking and security frameworks

Key Skills

Education

Bachelor's degree