Description

You will design, implement, and manage data pipelines to extract, transform, and load data from batch and streaming sources.

Responsibilities

• Build and manage data pipelines using Python, PySpark, and SQL.
• Develop and maintain efficient data models and analytical databases using Amazon Redshift or Snowflake.
• Administer databases via Amazon RDS or Redshift, focusing on schema design and performance optimization.
• Automate deployment and scaling of pipelines using infrastructure as code tools like Terraform or AWS CloudFormation.
• Implement ETL best practices including data validation, error handling, and quality checks.

Required Skills

• 2 years of experience in data engineering.
• Proficiency in Python, PySpark, and SQL.
• Hands-on experience with AWS DMS, AWS Glue, and AWS Step Functions.
• Experience managing Amazon S3, Amazon Redshift, and Amazon RDS.
• Working knowledge of Amazon EMR and AWS Lambda.
• Experience with AWS IAM for security and access management.
• Ability to optimize query performance and tune databases.
• Establish monitoring and logging mechanisms using AWS CloudWatch or Amazon CloudTrail.

Preferred Skills

• Experience ensuring data security and compliance with standards such as GDPR or HIPAA.

Key Skills

Education

Any Gradute