Description

Build and deploy components for a large-scale big data platform handling billions of daily messages.

Responsibilities

• Develop and optimize streaming data pipelines using Spark Structured Streaming and Kafka.
• Implement backend microservices in Azure using event-driven and domain-driven architectures.
• Design high-availability solutions for data ingestion from millions of devices.
• Observe, test, debug, and correct faults within the codebase.
• Produce technical specifications, documentation, and manage CI/CD processes.

Required Skills

• 9+ years of hands-on development experience.
• Expertise in Spark, Spark Streaming, and Spark Structured Streaming.
• Proficiency in Scala and Python.
• Strong experience with Kafka and micro-batching.
• Deep knowledge of Azure services: Databricks, Event Hubs, Stream Analytics, Data Explorer, and Monitors.
• Experience reading from and writing to RDBMS and NoSQL databases.
• Practical experience with CI/CD, code coverage, and integration testing.
• Bachelor's degree.

Preferred Skills

• Experience with Kubernetes.
• Knowledge of Yugabyte, Hadoop, and HDFS.
• Full data management skills including profiling, transformation, optimization, and governance.

Key Skills

Education

Bachelor's degree