Description

Key Skills: AWS Glue, AWS Lambda, S3, Redshift, Hadoop, PySpark, Data Migration, ETL, AI Code Assist Tools, Communication

Roles & Responsibilities:

• Design, develop and maintain scalable ETL pipelines using AWS Glue, PySpark and AWS Lambda for data processing
• Migrate structured and semi-structured data from existing data lakes such as S3 and Hadoop to Amazon Redshift
• Optimize data ingestion and transformation processes for performance and cost-efficiency to meet business requirements
• Collaborate with data architects, analysts and business stakeholders to understand data requirements and ensure data quality
• Implement data validation, cleansing and transformation logic to maintain data integrity and accuracy
• Monitor and troubleshoot data pipelines and workflows to ensure smooth operation and minimal downtime
• Ensure data security and compliance with organizational and regulatory standards for data governance
• Document data flows, architecture and processes to maintain comprehensive system documentation and knowledge transfer

Experience Required: Senior level position with experience in AWS data engineering and working knowledge of implementing data migration projects

Key Skills

Education

Any Graduate