Description

You will architect and build complex data integration solutions.

Responsibilities

• Design and implement data flow solutions using NiFi, defining sources, destinations, and transformations.
• Manage data ingestion from various sources (files, databases, APIs) and transform data for downstream consumption.
• Configure NiFi to route data and enrich it with necessary context like timestamps.
• Ensure data flows are scalable and performant under varying data volumes.
• Develop and maintain CI/CD pipelines for Airflow DAG deployment and version control.

Required Skills

• 10+ years of professional experience in data engineering or a related field.
• Strong expertise in NiFi, including processors, controllers, and data flow management.
• Proficiency in Java or Python for data processing tasks.
• Experience with ETL processes and data modeling concepts.
• Solid understanding of SQL and working with various databases.
• Knowledge of big data technologies, specifically Kafka.
• Familiarity with orchestration tools like Airflow, including executor management.
• Experience integrating with cloud services such as Azure Data Factory.
• Proficiency in managing CI/CD Pipelines.

Key Skills

Education

Any Gradute