Description

You will design and maintain enterprise-scale data pipelines using Apache Airflow and Apache NiFi. You own the full lifecycle from ingestion to storage, ensuring high availability and security.

Responsibilities

• Design and develop reusable DAGs in Apache Airflow to automate data workflows, scheduling, and monitoring.
• Create and optimize high-volume data streams using Apache NiFi, including custom processors and controller services.
• Develop and maintain Apache Spark jobs for large-scale data transformations, integrating them into orchestration layers.
• Manage object storage solutions using MinIO and AWS S3 for efficient data retrieval and high availability.
• Secure NiFi clusters and data flows using TLS/mTLS, managing Java KeyStores and SSL certificate lifecycles.

Required Skills

• 10+ years of experience in data engineering or platform development.
• Strong proficiency in Python and Java for pipeline development and custom NiFi processors.
• Deep expertise with Apache Airflow for DAG design and orchestration.
• Extensive experience with Apache NiFi, including troubleshooting the core framework.
• Proficiency in Apache Spark for heavy-duty data transformations.
• Strong Unix/Linux skills, including shell scripting for automation on RHEL/Ubuntu environments.
• Experience deploying and managing infrastructure on AWS, utilizing IAM for access control.

Preferred Skills

• Experience with MinIO and AWS S3 integration in hybrid data pipelines.
• Knowledge of establishing robust logging and alerting for 99.9% pipeline reliability.

Key Skills

Education

Any Gradute