Description

⦁ Design, deploy, and operate highly available Kafka clusters (on-prem, cloud, and/or managed services such as Confluent Cloud or AWS MSK).

⦁ Manage topics, partitions, quotas, retention policies, and consumer group strategies for performance and cost.

⦁ Own upgrades, patches, and migrations.

⦁ Implement and manage Kafka components: Kafka Connect, Schema Registry, MirrorMaker/Confluent Replicator, REST Proxy; familiarity with Kafka Streams and ksqlDB is a plus.

⦁ Performance tuning (producers/consumers, batching, compression, acks, ISR, controller health), throughput testing, and benchmarking.

⦁ Capacity planning, partitioning strategy, and cluster right-sizing.

Contract Deliverables

⦁ Hardware refresh plan: capacity model, sizing, architecture diagrams, migration/cutover strategy, risk register

⦁ Implement and validated on-prem clusters on refreshed hardware with performance benchmarks

⦁ Operational documentation: standards, runbooks, monitoring/alerts configuration, backup/restore and DR playbooks.

⦁ Knowledge transfer sessions and documentation handoff at milestones and project close.

Minimum Qualifications

⦁ 5+ years in systems/platform engineering, SRE, or DevOps; 4+ years operating Kafka in production at scale.

⦁ Deep knowledge of Kafka internals: partitions, replication, retention/compaction, rebalance strategies.

⦁ Hands-on with Kafka Connect, Schema Registry, MirrorMaker/Confluent Replicator.

⦁ Strong Linux fundamentals; networking (TCP, DNS, load balancing), and performance analysis.

⦁ Proficiency in automation/scripting.

⦁ Monitoring/observability: Data Dog, Grafana, JMX exporters, and log aggregation.

⦁ Experience with DR, multi-region design, and incident management.

⦁ Proven ability to produce clear, comprehensive documentation

Preferred Qualifications

⦁ Experience with Apache Kafka and AWS MSK operations and integration.

⦁ Experience executing hardware refreshes mor major cluster rebuilds/migrations with minimal downtime

Key Skills

Education

Bachelor's degree