Description

Position Overview : As a Kafka Architect at ValueLabs , you will play vital role in conceptualizing , designing , and implementing holistic and intuitive solutions for complex enterprise applications. Leveraging your expertise in user-centric design principles and your proficiency in cutting-edge technologies, you will collaborate with cross-functional teams to deliver exceptional user experiences across our suite of CRM an HR software products.

Key Responsibilities:

Confluent Kafka Infrastructure Architecture (3+ Years): • Architect and design enterprise-scale Confluent Kafka infrastructure supporting high-throughput, low-latency event streaming requirements • Design Kafka cluster topologies including multi-region, multi-datacenter, and hybrid cloud deployments • Plan and implement Kafka broker sizing, partition strategies, replication factors, and cluster scaling patterns • Architect Confluent Platform components: Confluent Server, ZooKeeper/KRaft, Schema Registry, Kafka Connect, ksqlDB, Confluent Control Center • Design high-availability and disaster recovery strategies for Kafka clusters including backup, replication, and failover mechanisms • Implement Kafka security architectures including SSL/TLS encryption, SASL authentication, and ACL-based authorization • Optimize Kafka infrastructure for performance, throughput, and cost-efficiency across cloud and on-premises environments • Evaluate and select deployment options: Confluent Cloud, Confluent Platform on-premises, or self-managed Kafka on cloud infrastructure Kafka Integration Solutions Design (5+ Years): • Design end-to-end event streaming solutions using Kafka for real-time data integration and event-driven architectures • Architect Kafka Connect solutions for integrating with databases, cloud storage, SaaS applications, and legacy systems • Design stream processing applications using Kafka Streams, ksqlDB, or Apache Flink for real-time data transformation • Implement event sourcing and CQRS (Command Query Responsibility Segregation) patterns using Kafka as the event store • Design microservices integration patterns using Kafka for asynchronous communication and event-driven choreography • Architect data pipeline solutions for ETL/ELT, data synchronization, and real-time analytics use cases • Design schema management strategies using Confluent Schema Registry with Avro, Protobuf, and JSON Schema • Implement exactly-once processing semantics and transactional messaging patterns for critical business applications Performance Optimization & Tuning: • Optimize Kafka cluster performance: throughput tuning, latency reduction, and resource utilization • Tune Kafka producer and consumer configurations for optimal performance and reliability • Design and implement partitioning strategies for balanced load distribution and parallel processing • Monitor and optimize Kafka consumer lag, rebalance operations, and consumer group management • Implement Kafka compaction strategies and log retention policies for storage optimization • Tune JVM settings, garbage collection, and OS-level configurations for Kafka brokers • Conduct performance benchmarking and capacity planning for future growth Security & Governance: • Architect comprehensive Kafka security models including authentication, authorization, and encryption • Implement SSL/TLS encryption for data in transit and encryption at rest for sensitive data • Design SASL-based authentication mechanisms: SASL/PLAIN, SASL/SCRAM, SASL/GSSAPI (Kerberos), SASL/OAUTHBEARER • Implement fine-grained ACLs and role-based access control for Kafka resources • Design data governance frameworks for event streaming including data lineage, quality, and compliance • Ensure compliance with regulatory requirements: GDPR, PCI-DSS, HIPAA, SOX for streaming data • Implement audit logging and monitoring for security events and access patterns Platform Operations & Reliability: • Design monitoring and observability solutions for Kafka infrastructure using Confluent Control Center, Prometheus, Grafana • Implement alerting mechanisms for cluster health, consumer lag, and critical metrics • Create runbooks and operational procedures for Kafka cluster maintenance and incident response • Design automated failover and self-healing mechanisms for Kafka infrastructure • Plan and execute Kafka upgrades, migrations, and platform maintenance with minimal downtime • Implement disaster recovery strategies including MirrorMaker 2.0 for cross-cluster replication • Conduct chaos engineering and resilience testing for Kafka infrastructure Integration Architecture: • Design integration patterns between Kafka and enterprise systems: databases, data lakes, data warehouses • Architect Kafka-to-Kafka replication strategies for multi-region and multi-cloud scenarios • Design integration with cloud-native services: AWS Kinesis, Azure Event Hubs, Google Pub/Sub • Implement CDC (Change Data Capture) patterns using Debezium or similar tools with Kafka • Design API integration patterns using Kafka as the messaging backbone • Architect hybrid cloud and multi-cloud Kafka deployments with data sovereignty considerations Development & Best Practices: • Establish Kafka development standards, coding guidelines, and best practices for engineering teams • Design and review Kafka producer and consumer applications for performance and reliability • Implement error handling, dead letter queues, and retry mechanisms for robust event processing • Create reusable Kafka components, libraries, and templates for common patterns • Establish testing strategies for Kafka applications including unit, integration, and chaos testing • Mentor development teams on Kafka patterns, anti-patterns, and optimization techniques • Conduct code reviews and architecture reviews for Kafka-based solutions Stakeholder Collaboration: • Collaborate with data engineers, software engineers, and architects to design streaming solutions • Work with security and compliance teams to ensure Kafka implementations meet regulatory requirements • Partner with infrastructure and DevOps teams for Kafka deployment automation and CI/CD integration • Engage with business stakeholders to understand streaming requirements and translate to technical solutions • Present Kafka architecture designs, roadmaps, and recommendations to technical and executive audiences • Contribute to internal communities of practice and knowledge sharing sessions Innovation & Strategy: • Research and evaluate emerging technologies in the event streaming and real-time analytics space • Develop Kafka platform roadmaps and migration strategies for legacy integration systems • Explore new Confluent Platform features and capabilities for potential adoption • Drive adoption of event-driven architecture patterns and streaming best practices • Identify opportunities for automation, optimization, and cost reduction in Kafka infrastructure • Contribute to open-source Kafka community and stay current with Kafka ecosystem developments PRIMARY SKILLS (REQUIRED) Confluent Kafka Infrastructure (3+ Years) • Expert-level experience architecting and implementing large-scale Confluent Kafka infrastructure including: - Confluent Platform architecture: Confluent Server, Schema Registry, Kafka Connect, ksqlDB, Control Center - Kafka cluster sizing, topology design, and capacity planning for high-throughput scenarios - Multi-region and multi-datacenter Kafka deployments with disaster recovery - ZooKeeper and KRaft (Kafka Raft) metadata management modes - Kafka broker configuration, tuning, and performance optimization - Partitioning strategies, replication factors, and ISR (In-Sync Replicas) management - Kafka security: SSL/TLS, SASL mechanisms, ACLs, encryption at rest - Confluent Cloud architecture and hybrid cloud strategies - Cluster upgrades, migrations, and maintenance procedures Kafka Integration Solutions (5+ Years): • Extensive experience designing and implementing Confluent Kafka integration solutions including: - Event-driven architecture patterns: event sourcing, CQRS, saga pattern, outbox pattern - Kafka Connect framework for source and sink connectors (JDBC, S3, Elasticsearch, etc.) - Kafka Streams API for stream processing and real-time data transformation - ksqlDB for SQL-based stream processing and real-time analytics - Schema Registry integration with Avro, Protobuf, and JSON Schema - Exactly-once semantics and transactional messaging - Microservices integration using Kafka for asynchronous communication - Real-time data pipelines for ETL/ELT and data synchronization - CDC (Change Data Capture) patterns with Debezium and Kafka Connect - Multi-tenant Kafka architectures and data isolation strategies Kafka Performance & Optimization: • Deep expertise in Kafka performance tuning: - Producer and consumer configuration optimization - Consumer group management and rebalancing strategies - Consumer lag monitoring and mitigation techniques - JVM tuning and garbage collection optimization for Kafka brokers - OS-level tuning: network, disk I/O, memory management - Throughput and latency optimization strategies - Storage optimization and log compaction strategies Stream Processing: • Advanced knowledge of stream processing frameworks: - Kafka Streams DSL and Processor API - ksqlDB for continuous SQL queries and stream processing - Windowing operations: tumbling, hopping, session windows - Stateful stream processing and interactive queries - Joins, aggregations, and transformations in real-time streams - Apache Flink integration with Kafka for complex event processing Security & Governance: • Comprehensive Kafka security expertise: - SSL/TLS certificate management and PKI for Kafka - SASL authentication mechanisms and configuration - ACL-based authorization and resource-level permissions - Data encryption at rest and in transit - Audit logging and security monitoring - Data governance and compliance for streaming data Monitoring & Observability: • Kafka monitoring and observability tools: - Confluent Control Center for cluster monitoring and management - JMX metrics and Kafka metrics export - Prometheus and Grafana for metrics collection and visualization - Distributed tracing for Kafka applications (OpenTelemetry, Jaeger) - Consumer lag monitoring and alerting - Log aggregation and analysis (ELK stack, Splunk) Programming & Scripting: • Strong programming skills in: - Java for Kafka client applications and Kafka Streams - Python for Kafka integration and automation scripts - SQL for ksqlDB and stream processing queries - Bash/Shell scripting for operational automation - Go or Scala (preferred) for Kafka ecosystem tools Cloud Platforms: • Cloud platform expertise for Kafka deployments: - AWS: MSK (Managed Streaming for Kafka), EC2, S3, IAM - Azure: Event Hubs for Kafka, Azure Kubernetes Service (AKS), Azure Storage - Google Cloud: Pub/Sub, Cloud Storage, GKE - Confluent Cloud managed service architecture and configuration SECONDARY SKILLS (PREFERRED) Additional Streaming Technologies: • Apache Pulsar for multi-tenant messaging and geo-replication • AWS Kinesis for cloud-native streaming • Azure Event Hubs and Stream Analytics • Google Pub/Sub and Dataflow • Apache Flink for complex event processing • Apache Spark Streaming for micro-batch processing • NATS or RabbitMQ for alternative messaging patterns Data Integration & Storage: • Database technologies: PostgreSQL, MySQL, MongoDB, Cassandra, Elasticsearch • Data lakes: Apache Hadoop, Apache Iceberg, Delta Lake • Data warehouses: Snowflake, Databricks, BigQuery, Redshift, Azure Synapse • Cloud storage: S3, Azure Blob Storage, Google Cloud Storage • Search and analytics: Elasticsearch, OpenSearch, ClickHouse Microservices & Containerization: • Kubernetes and container orchestration for Kafka deployments • Helm charts and operators for Kafka on Kubernetes (Strimzi, Confluent Operator) • Service mesh: Istio, Linkerd for microservices communication • Docker containerization for Kafka applications • Microservices patterns: API Gateway, service discovery, circuit breakers DevOps & Automation: • CI/CD pipelines for Kafka application deployment • Infrastructure as Code: Terraform, Ansible, CloudFormation for Kafka infrastructure • GitOps workflows for Kafka configuration management • Configuration management tools: Puppet, Chef • Deployment automation and blue-green deployment strategies Data Governance & Quality: • Data lineage tracking and metadata management • Data quality frameworks for streaming data • Master data management (MDM) in event-driven contexts • Data catalog tools: Apache Atlas, Collibra, Alation • GDPR and data privacy compliance for real-time data Soft Skills • Architecture documentation and diagramming (C4 model, ArchiMate) • Technical leadership and thought leadership • Stakeholder management and communication • Mentoring and knowledge transfer • Problem-solving in complex distributed systems • Continuous learning and technology evaluation

Qualifications:

We are seeking a highly skilled and experienced Senior Integration Developer to join our dynamic team. The ideal candidate will be responsible for designing, developing, and maintaining integration solutions between various enterprise systems. This role requires a strong background in Java-based backend development, API development, and extensive experience with Confluent Kafka and SnapLogic. The Senior Integration Developer will play a crucial role in building and supporting our integration platform, ensuring seamless data flow and connectivity across the organization

Key Skills

Education

Any Graduate