Description

You will build and maintain data processing pipelines using Spark and Python.

Responsibilities

• Implement ETL processes to extract, transform, and load data from diverse sources.
• Design and develop both streaming and batch processing workflows.
• Optimize PySpark code for performance, scalability, and data integrity.
• Integrate data from multiple systems to create unified views for analysis.
• Collaborate with cross-functional teams to design data architecture and meet integration needs.

Required Skills

• 5+ years of experience in Big Data environments.
• Strong proficiency in Python and PySpark.
• Deep understanding of Spark/Big Data frameworks.
• Hands-on experience with ETL processes and pipeline development.
• Experience managing CI/CD workflows using Jenkins or GitHub.
• Ability to implement streaming workflows using PySpark Streaming.
• Experience developing batch processing workflows for large-scale data.

Preferred Skills

• Any Graduate degree.

Key Skills

Education

Any Graduate