Description

Design and develop scalable data pipelines on AWS for batch and near-real-time processing.

This role is hybrid.

Responsibilities

• Build ETL/ELT workflows using AWS Glue, orchestrated with Apache Airflow or AWS Step Functions.
• Develop data transformations using Python and PySpark, implementing complex SQL logic for analytics.
• Model data for analytics using dimensional modeling and support warehousing patterns in Amazon Redshift.
• Implement AWS security, access controls (IAM), and support reliable operations for data services.

Required Skills

• 7-10 years of data engineering experience on AWS, including core services (Glue, S3, Redshift, Athena, Lambda).
• Strong proficiency in Python and PySpark for data processing.
• Expert SQL skills with deep understanding of database concepts and dimensional modeling.
• Experience with orchestration tools like Apache Airflow and/or AWS Step Functions.
• Familiarity with IAM security fundamentals and data warehousing architectures.
• Proficiency in CI/CD practices and DevSecOps.

Preferred Skills

• AWS Certified Data Engineer or Solutions Architect certification.
• Experience with Infrastructure as Code (Terraform) and streaming technologies (Kafka, Kinesis).
• Exposure to Databricks or Snowflake.

Key Skills

Education

Any Graduate