Description

You will design and maintain data pipelines to support large-scale data processing and analysis.

Responsibilities

• Implement ETL processes to extract, transform, and load data from various sources.
• Design and implement streaming workflows using PySpark Streaming and batch processing workflows.
• Develop efficient data processing and analysis scripts using Python and PySpark.
• Collaborate with cross-functional teams and data analysts to design architecture and meet integration needs.
• Ensure data quality, integrity, and consistency throughout the entire pipeline.

Required Skills

• 3+ years of experience in data engineering roles.
• Proficiency in Python.
• Strong experience with PySpark.
• Deep understanding of ETL processes.
• Experience with both streaming and batch workflows.
• Knowledge of CI/CD tools like Jenkins or GitHub.
• Bachelor's degree in a relevant field.

Key Skills

Education

Bachelor's degree