Description

You will design and implement high-performance, event-driven data pipelines for a large credit card processing system, ensuring low-latency processing and high availability.

This role is on-site.

Responsibilities

• Create complex Ab Initio continuous flow graphs with stateful joins, sliding time windows, and aggregations.
• Implement event-driven pipelines using Kafka, message queues, and file streams for near-real-time updates.
• Ensure resilience of continuous flows through checkpointing and recovery to guarantee exactly-once processing.
• Develop and monitor ETL pipelines for batch and real-time data ingestion, validation, and transformation.
• Troubleshoot performance issues in large-scale clusters and maintain 24/7 production reliability.

Required Skills

• Deep knowledge of Ab Initio ecosystem: GDE, EME, Conduct>It, and Continuous Flows.
• Proficiency in Unix/Linux shell scripting, SQL, and database technologies (Oracle, Teradata).
• Experience with Kafka, message queues, and real-time stateful services.
• Understanding of data security, governance, and compliance standards.
• Experience with cloud-based big data platforms and containerization (Docker, Kubernetes).
• Knowledge of CI/CD pipelines for data engineering projects.
• 5+ years of experience in data engineering or related field.

Preferred Skills

• Proficiency in Java, Scala, or Python.
• Experience with business process modeling and documentation tools.

Key Skills

Education

Any Graduate