Description

You will design and implement high-performance data processing engines using Apache Spark and Scala, applying advanced algorithms to solve complex data problems.

Responsibilities

• Design scalable, reusable data processing systems using Spark SQL, DataFrames, and RDDs for batch and real-time workloads.
• Implement complex algorithms including search trees (AVL, red-black), clustering (K-means, DBSCAN), and graph algorithms (PageRank, Dijkstra’s).
• Optimize Spark jobs for performance, ensuring high availability and efficient handling of structured, semi-structured, and unstructured data.
• Apply dynamic programming and greedy algorithms to solve optimization problems such as knapsack and shortest paths.
• Collaborate with data scientists to define architectural best practices and implement end-to-end data solutions.

Required Skills

• Strong proficiency in Scala with deep expertise in functional and object-oriented programming principles.
• Hands-on experience with Apache Spark ecosystem, including optimization of distributed data processing.
• Deep understanding of data structures: heaps, hash tables, and advanced trees.
• Proven ability to implement clustering algorithms (K-means, GMM, hierarchical) for large-scale datasets.
• Experience with graph algorithms and network analysis techniques.
• Strong foundation in sorting, searching, and dynamic programming for computational efficiency.
• Minimum 7 years of professional experience in data engineering or algorithmic development.

Key Skills

Education

Any Graduate