Description

You will design, develop, and optimize data pipelines using Spark and Python within AWS environments.

Responsibilities

• Design and develop optimized Spark ETL pipelines.
• Build and manage Airflow DAGs for workflow orchestration.
• Write complex SQL queries and optimize database performance.
• Implement data ingestion and transformation processes using Python libraries.
• Manage data workflows within AWS EMR, Athena, S3, and AWS Glue.

Required Skills

• 3-4 years of experience building optimized ETL pipelines.
• 3+ years of experience building Spark ETL pipelines.
• Proficiency with Python and its data transformation libraries.
• Strong command of SQL and query optimization.
• Hands-on experience with Spark or PySpark.
• Experience with AWS EMR, Athena, S3, and AWS Glue.
• Experience managing Airflow DAGs.

Preferred Skills

• Any Graduate degree.

Key Skills

Education

Any Graduate