Description

Build and maintain large-scale data processing solutions for distributed systems.

This role is on-site.

Responsibilities

• Design, develop, and maintain ETL pipelines using PySpark and AWS Glue.
• Orchestrate and schedule data workflows using Apache Airflow.
• Optimize data processing jobs for performance and cost-efficiency.
• Ensure data quality and consistency across large datasets from various sources.
• Monitor pipeline health and proactively resolve data-related issues.

Required Skills

• 8+ years of experience in data engineering roles.
• Strong expertise in PySpark for distributed data processing.
• Hands-on experience with AWS Glue, S3, Athena, and Lambda.
• Experience with Apache Airflow for workflow orchestration.
• Strong proficiency in SQL for data extraction and transformation.
• Familiarity with data modeling and data lake/warehouse architectures.
• Experience with Git and CI/CD processes.
• Ability to write clean, scalable, production-grade code.

Preferred Skills

• Experience collaborating with Data Scientists and Analysts.

Key Skills

Education

Any Gradute