Description

You will own the design, development, and maintenance of high-volume data pipelines and infrastructure.

Responsibilities

• Design and develop reusable DAGs in Apache Airflow to automate data workflows and monitoring.
• Build and optimize high-volume data streams using Apache NiFi, including custom processors and controller services.
• Develop and maintain Apache Spark jobs for large-scale data transformations and integrate them into orchestration layers.
• Implement and manage object storage solutions using MinIO and AWS S3 for efficient data retrieval.
• Secure NiFi clusters using TLS/mTLS, managing Java KeyStores and SSL certificate lifecycles.

Required Skills

• 10+ years of experience in data engineering roles.
• Proficiency in Apache Airflow for pipeline orchestration.
• Deep experience with Apache NiFi, including custom processor development.
• Strong Java programming skills for NiFi customization and framework troubleshooting.
• Experience with Python for scripting and automation.
• Hands-on AWS skills, including IAM, S3, and cloud-native service integration.
• Proficiency in Unix/Linux system administration and shell scripting.
• Knowledge of Apache Spark for big data processing.

Preferred Skills

• Experience with MinIO object storage management.
• Background in establishing logging and alerting for 99.9% pipeline reliability.

Key Skills

Education

Bachelor's degree