Description

You will build and maintain large-scale data processing solutions.

Responsibilities

• Design, develop, and maintain ETL pipelines using PySpark and AWS Glue.
• Orchestrate and schedule data workflows using Apache Airflow.
• Optimize data processing jobs for performance and cost-efficiency.
• Ensure data quality and consistency when working with large datasets from various sources.
• Monitor data pipeline health and resolve data-related issues proactively.

Required Skills

• 8+ years of experience in data engineering roles.
• Strong expertise in PySpark for distributed data processing.
• Hands-on experience with AWS Glue and AWS services (S3, Athena, Lambda).
• Experience with Apache Airflow for workflow orchestration.
• Strong proficiency in SQL for data extraction and transformation.
• Familiarity with data modeling and data lake/data warehouse architectures.
• Experience with Git and CI/CD processes.
• Ability to write clean, scalable, production-grade code.

Key Skills

Education

Any Gradute