Description

You will own the design and operation of data ingestion and transformation pipelines on AWS.

Responsibilities

• Develop and maintain PySpark-based ETL pipelines for batch and incremental data processing.
• Build and operate AWS Glue Spark jobs (batch and event-driven), handling configuration, scaling, retrieval, and cost optimization.
• Design and maintain event-driven data workflows triggered by S3, EventBridge, or streaming sources.
• Design and implement dimensional data models (star/snowflake schemas), including Fact and dimension tables and Slowly Changing Dimensions (SCDs).
• Monitor, troubleshoot, and optimize data pipelines for reliability and performance.

Required Skills

• 12+ years of professional experience in data engineering.
• Expertise in PySpark for ETL development.
• Strong experience with AWS services, particularly AWS Glue and S3.
• Proficiency in Spark for large-scale data processing.
• Experience designing and optimizing data models for Amazon Redshift, including distribution and sort keys.
• Ability to manage Glue Catalog and schema definitions.
• Experience implementing data quality controls and managing grain definitions.

Key Skills

Education

Any Gradute