Description

You will design and manage data pipelines and ETL processes within an AWS environment.

Responsibilities

• Design, implement, and optimize automated data pipelines using PySpark, AWS Glue, and AWS Data Pipeline.
• Develop and maintain ETL processes to efficiently process large datasets.
• Build and manage automated data workflows for seamless scheduling and monitoring.
• Integrate data across various AWS storage services including S3, Redshift, and RDS.
• Optimize and scale pipelines for performance and cost efficiency using Lambda, S3, and EC2.

Required Skills

• 6+ years of experience in data engineering within cloud environments.
• Proficiency in Python and PySpark for data processing and transformation.
• Hands-on experience with AWS Glue for ETL jobs and workflow management.
• Experience with AWS Data Pipeline (DPL) for orchestration.
• Strong knowledge of SQL for querying relational and semi-structured data.
• Practical experience with AWS S3, Lambda, Redshift, RDS, and EC2.
• Understanding of ETL concepts, data warehousing, and Big Data technologies.
• Experience with Git for version control and CI/CD pipelines.
• Familiarity with Docker and containerization.
• Any Graduate degree.

Preferred Skills

• Experience with Amazon EMR, Kinesis, and Data Lakes.
• Knowledge of data governance, including data lineage and auditing.

Key Skills

Education

Any Graduate