Description

You will architect and build complex data integration solutions, owning the design, implementation, and reliability of data flow systems.

This role is on-site.

Responsibilities

• Design and implement data flow solutions using NiFi, defining sources, destinations, and transformations.
• Manage data ingestion from files, databases, and APIs, transforming data for downstream consumption.
• Configure NiFi to route data and enrich it with necessary context like timestamps.
• Ensure data flows are scalable and performant under varying data volumes.
• Develop and maintain CI/CD pipelines for Airflow DAG deployment and version control.

Required Skills

• 10+ years of professional experience in data engineering or a related field.
• Strong expertise in NiFi, including processors, controllers, and data flow management.
• Proficiency in Java or Python for data processing tasks.
• Experience with ETL processes and data modeling concepts.
• Solid understanding of SQL and working with various databases.
• Knowledge of big data technologies, specifically Kafka.
• Familiarity with orchestration tools like Airflow, including executor management.
• Experience integrating with cloud services such as Azure Data Factory.
• Proficiency in managing CI/CD Pipelines.

Preferred Skills

• Experience with Kubernetes for container orchestration.

Key Skills

Education

Any Gradute