Description

You will design, build, and maintain scalable data pipelines for the ecosystem.

This role is hybrid.

Responsibilities

• Develop and maintain batch and streaming data pipelines using PySpark.
• Implement complex business logic utilizing Spark SQL, DataFrame, and RDD APIs.
• Design and manage infrastructure using Infrastructure as Code (Terraform).
• Operate and optimize ETL/ELT processes across cloud data platforms.
• Build and manage data pipelines covering both batch and streaming patterns.

Required Skills

• 6+ years of professional experience in data engineering.
• Strong proficiency in Python with solid fundamentals in data structures and algorithms.
• Expertise in PySpark, including Spark SQL and DataFrame API usage.
• Hands-on experience with AWS services: S3, IAM, Lambda, Glue ETL, EMR, and Athena.
• Experience with infrastructure automation using Terraform.
• Familiarity with distributed data processing frameworks and cloud data warehousing or data lake concepts.

Preferred Skills

• Experience with Angular.
• Familiarity with API Gateway.
• Networking knowledge.

Key Skills

Education

Any Graduate