Description

Build and manage data pipelines using PySpark and Scala within an AWS environment.

Responsibilities

• Develop and maintain ETL processes to support data engineering workflows.
• Build and orchestrate data pipelines using Airflow.
• Optimize SQL queries and data structures for efficiency.
• Manage data streaming and processing tasks using Apache Kafka.

Required Skills

• 3+ years of hands-on experience with AWS Cloud.
• 3+ years of expertise in PySpark and Data Engineering.
• Strong proficiency in SQL technologies.
• Experience building and managing ETL pipelines.
• Experience with Airflow for workflow orchestration.
• Proficiency in Python.
• Experience with Hive.
• Experience with Apache Kafka.

Key Skills

Education

Any Graduate