Description

Lead the architecture, design, and implementation of Apache Airflow workflows and data pipelines.

Responsibilities

• Architect and design Apache Airflow implementations, including cluster installation, configuration, and monitoring.
• Build, maintain, and orchestrate data pipelines while ensuring performance tuning of DAGs and tasks.
• Develop guidelines for Airflow clusters and DAGs to establish implementation best practices.
• Provide SME support for Airflow administration and integration within the platform ecosystem.
• Guide the team through data pipeline development, change management, and in-house knowledge transfer.

Required Skills

• 8 to 12 years of professional experience, including 3 to 5 years as an Apache Airflow developer or architect.
• Expertise in Apache Airflow, including DAG development, REST services, and cluster management.
• Hands-on experience with Kafka and Azure Synapse.
• Strong proficiency in Python and its frameworks.
• Working knowledge of Azure and relational databases with SQL query authoring skills.
• Experience building and optimizing big data pipelines, architectures, and datasets.
• Proficiency in modern programming languages such as Java, C#, JavaScript, or TypeScript.
• Experience with CI-CD tools like Jenkins and GitHub code repositories.
• Solid understanding of software engineering practices, including code reviews, source control, and the full SDLC.
• Experience working within Agile/Scrum principles.

Preferred Skills

• Experience with Azure Kubernetes Service (AKS), Databricks, or Jenkins.

Key Skills

Education

ANY GRADUATE