Description

You will architect and build data flow solutions utilizing NiFi and Airflow for complex ETL requirements.

Responsibilities

• Design and implement data flow solutions using NiFi, defining sources, destinations, and transformations.
• Manage data ingestion from various sources (files, databases, APIs) and transform it for downstream consumption.
• Configure NiFi to route and enrich data with necessary metadata.
• Ensure data flows maintain scalability and performance under varying data volumes.
• Develop and maintain CI/CD pipelines for Airflow DAG deployment and version control.

Required Skills

• 10+ years of professional experience in relevant engineering roles.
• Deep expertise in NiFi architecture, processors, controllers, and data flow management.
• Proficiency in Java or Python for scripting and development.
• Strong experience with ETL processes and data modeling concepts.
• Familiarity with SQL and database interaction for data manipulation.
• Knowledge of Big Data technologies, specifically Kafka.
• Experience with Airflow architecture, including scheduler and executor concepts.
• Familiarity integrating workflows with cloud services like Azure Data Factory.
• Experience with CI/CD practices.

Key Skills

Education

Any Gradute