Description

You will design and implement high-performance, event-driven data pipelines for a large-scale credit card processing system, ensuring low-latency processing and high availability.

This role is on-site.

Responsibilities

• Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and aggregations.
• Implement event-driven data pipelines using Kafka, message queues, and file streams.
• Ensure resilience of continuous flows through checkpointing and recovery to guarantee exactly-once processing.
• Apply advanced Ab Initio components (Reformat, Rollup, Join, Partition) to optimize low-latency performance.
• Monitor live production streams for 24/7 reliability, troubleshoot data issues, and automate workflows using scripting.

Required Skills

• Deep knowledge of Ab Initio ecosystem: GDE, EME, Conduct>It, and Continuous Flows.
• Proficiency in Unix/Linux shell scripting, SQL, and database technologies (Oracle, Teradata).
• Experience with Kafka, message queues, and real-time stateful services.
• Understanding of credit card processing systems and data security governance.
• 5+ years of experience in data engineering or ETL pipeline development.

Preferred Skills

• Experience in Java, Scala, or Python.
• Familiarity with containerization (Docker, Kubernetes) and CI/CD pipelines.
• Experience with cloud-based big data platforms.

Key Skills

Education

Any Graduate