Description

You will design, implement, and maintain scalable Kafka-based data streaming pipelines.

Responsibilities

• Design robust and scalable Kafka-based data pipelines considering throughput, fault tolerance, and latency.
• Implement and manage data streaming pipelines using Confluent Kafka architecture.
• Troubleshoot Kafka cluster health, including broker failures, consumer lag, and message delivery issues.
• Diagnose performance bottlenecks related to throughput, network congestion, or disk I/O.
• Manage data schema evolution and serialization formats.

Required Skills

• 5+ years of experience in Kafka development and messaging.
• Deep understanding of Confluent Kafka architecture, including brokers, topics, partitions, and consumer groups.
• Proficiency in Java or Scala; familiarity with Python.
• Strong experience with Kafka Connect and KSQL architecture.
• Hands-on experience with cloud platforms including AWS, Azure, or GCP.
• Knowledge of data serialization formats such as JSON, Avro, and Protobuf.
• Experience with Kafka security mechanisms, including authentication, authorization, and encryption.
• Ability to perform log analysis and monitor metrics like consumer lag and broker CPU usage.
• Solid understanding of distributed systems concepts like leader election and fault tolerance.

Preferred Skills

• Knowledge of Kafka Streams API for real-time data processing.

Key Skills

Education

Any Gradute