Description

You will design and implement scalable data streaming pipelines using Confluent Kafka architecture.

This role is remote.

Responsibilities

• Design robust Kafka-based data pipelines considering throughput, fault tolerance, and latency.
• Implement and manage data streaming pipelines using Kafka Connect and KSQL architecture.
• Troubleshoot cluster issues including broker failures, consumer lag, and message delivery discrepancies.
• Monitor Kafka cluster health, tracking metrics like message throughput, broker CPU usage, and network latency.
• Optimize Kafka settings to resolve performance bottlenecks related to network congestion or disk I/O.

Required Skills

• 6+ years of experience in distributed systems and data management.
• Deep knowledge of Confluent Kafka architecture, including brokers, topics, partitions, and consumer groups.
• Proficiency in Java or Scala for debugging custom producers and consumers.
• Experience with data serialization formats including JSON, Avro, and Protobuf.
• Hands-on experience with Kafka Connect and KSQL architecture.
• Ability to manage Kafka security mechanisms such as authentication, authorization, and encryption.
• Experience deploying and managing Kafka on cloud platforms including AWS, Azure, or GCP.
• Strong understanding of distributed systems concepts like leader election and fault tolerance.
• Competency in log analysis and network troubleshooting for Kafka connectivity.

Preferred Skills

• Knowledge of Kafka Streams API for real-time data processing.
• Proficiency in Python.

Key Skills

Education

Any Graduate