Description

You will build and maintain production-grade data pipelines using Python and PySpark.

Responsibilities

• Design, develop, and lead data engineering teams.
• Design, schedule, and monitor complex workflows using Airflow.
• Build and maintain CI/CD pipelines for data engineering projects.
• Implement automated unit and integration testing using TDD practices.

Required Skills

• 7+ years of overall IT experience with a strong Data Engineering background.
• 5+ years experience building production data pipelines with Python and PySpark.
• Strong expertise in Airflow for DAG management.
• Experience with GitHub, Docker, and cloud-native CI/CD tools.
• Hands-on experience with TDD and automated testing.
• Solid SQL experience for data ingestion and analytics.
• Proficiency in Git for version control (branching, merging, PRs).
• Experience in Unix/Linux environments with strong command-line skills.
• Experience with containerization (Docker) and cloud platforms (GCP preferred).

Preferred Skills

• Experience with OpenShift, Cloudera, Tableau, Confluence, and Jira.

Key Skills

Education

Bachelor's degree