Description

You will design and maintain data pipelines for large-scale processing and analysis.

Responsibilities

• Implement ETL processes to extract, transform, and load data from diverse sources.
• Design and implement streaming workflows using PySpark Streaming alongside batch processing workflows.
• Develop efficient data processing and analysis scripts using Python and PySpark.
• Collaborate with cross-functional teams and data analysts to define architecture and meet integration requirements.
• Ensure data quality, integrity, and consistency across the entire pipeline.

• 3+ years of experience in data engineering roles.
• Proficiency in Python.
• Strong experience with PySpark.
• Deep understanding of ETL processes.
• Experience with both streaming and batch workflows.
• Knowledge of CI/CD tools like Jenkins or GitHub.
• Bachelor's degree in a relevant field.

Key Skills

Education

Bachelor's degree