Description

Lead data engineering initiatives using PySpark, AWS Glue, and Redshift. Own the design and implementation of scalable ETL solutions.

Responsibilities

• Build and maintain ETL pipelines using PySpark, AWS Glue, and Redshift.
• Implement CI/CD workflows using AWS CodeBuild, CodeCommit, CodeDeploy, and CodePipeline.
• Develop reusable libraries and clean, maintainable Python code.
• Design scalable solutions leveraging S3, EC2, EMR, SNS, Lambda, Step Functions, and EventBridge.
• Implement automated testing using PyTest and manage code versioning with Git.

Required Skills

• 8+ years of experience in Data Engineering.
• 6+ years of hands-on experience with PySpark.
• 5+ years of experience with AWS Glue and AWS Redshift.
• 5+ years of experience with AWS CI/CD pipelines (CodeBuild, CodeCommit, CodeDeploy, CodePipeline).
• Strong proficiency in Python, Hadoop, and Spark.
• Experience implementing automated testing platforms like PyTest.
• Proficiency in understanding code versioning tools such as Git, SVN, or TFS.
• Bachelor's degree or higher in a relevant field.

Preferred Skills

• Experience with fundamental design principles for building scalable solutions.
• Ability to write reusable libraries and clean, commented code.

Key Skills

Education

Any Graduate