Description

You will own the design and implementation of high-throughput Kafka integration pipelines, ensuring data integrity and operational stability across distributed systems.

Responsibilities

• Tune Kafka producers and consumers to optimize throughput, latency, and resource utilization in production environments.
• Implement the outbox pattern and manage event replay and backfill processes to maintain data consistency.
• Configure and manage consumer groups, partitioning strategies, and exactly-once semantics for reliable message processing.
• Handle schema evolution via Schema Registry and manage dead letter queues for failed message handling.
• Integrate Kafka streams with PostgreSQL for transactional writes and set up comprehensive monitoring and alerting.

Required Skills

• 8+ years of professional experience as an Apache Kafka Integration Engineer or similar role.
• 4+ years of hands-on experience operating Apache Kafka or equivalent streaming platforms in production.
• Deep expertise in Kafka internals, including exactly-once semantics, consumer group management, and partitioning strategies.
• Proficiency with Schema Registry for managing schema evolution and compatibility.
• Experience implementing the outbox pattern and managing event replay/backfill mechanisms.
• Strong background in integrating streaming platforms with relational databases, specifically PostgreSQL.
• Ability to design and implement operational monitoring and alerting for Kafka clusters.

Preferred Skills

• Experience with distributed transactional systems and ensuring data consistency across microservices.
• Knowledge of additional streaming frameworks beyond Kafka (e.g., RabbitMQ, Pulsar).

Key Skills

Education

Any Graduate