Description

You will lead major data engineering initiatives, designing and deploying scalable data systems.

Responsibilities

• Design, develop, and optimize scalable data pipelines using Python and Py Spark for batch and streaming workloads.
• Build, schedule, and monitor complex workflows using Airflow, ensuring reliability.
• Architect and implement CI/CD pipelines for data engineering projects using GitHub, Docker, and cloud-native solutions.
• Apply test-driven development (TDD) practices and automate unit/integration tests for data pipelines.
• Mentor junior engineers on coding, testing, and deployment best practices.

Required Skills

• Minimum 5+ years of practical experience building production-grade data pipelines using Python and Py Spark.
• Proven track record of designing, deploying, and managing Airflow DAGs in enterprise environments.
• Ability to build and maintain CI/CD pipelines for data engineering workflows, including automated testing and deployment.
• Experience with containerization (Docker) and cloud platforms (GCP) for data engineering workloads.
• Proficiency with Git for source code management and collaboration (GitHub/GitLab workflows).
• Solid understanding of SQL for data ingestion and analysis.
• Strong command-line skills in Unix-like environments.
• Ability to write object-oriented Python code following industry best practices.
• Minimum of 7+ years overall IT experience.

Key Skills

Education

Any Graduate