Description

Own the architecture, design, and optimization of large-scale Hadoop-based data platforms for regulatory and surveillance use cases.

This role is hybrid.

Responsibilities

• Define target architecture and strategic roadmaps for surveillance data processing using Hadoop, Spark, PySpark, and Python.
• Translate business requirements into scalable technical designs for alert generation, enrichment, and reporting pipelines.
• Modernize and optimize large-scale Hadoop workloads to improve performance, stability, and control coverage.
• Lead architecture review forums, mentor senior engineers, and enforce standard architectural patterns.
• Maintain high-quality documentation for business requirements, data definitions, and process flows.

Required Skills

• 10+ years of experience in large-scale distributed environments.
• Deep expertise in PySpark with performance tuning and optimization.
• Strong Python development skills for distributed systems.
• Solid knowledge of the Hadoop ecosystem (HDFS, Hive/Impala, YARN).
• Proficiency in Java, Scala, and SQL.
• Experience designing data lakes, ELT/ETL pipelines, and batch/real-time processing solutions.
• Understanding of non-functional requirements and production support models.

Preferred Skills

• Financial services experience, particularly in Market Surveillance, AML, Fraud, or Risk Technology.
• Experience supporting regulatory or audit-facing platforms.
• Exposure to Kafka, Spark Structured Streaming, and orchestration tools like Airflow, Control-M, or Oozie.

Key Skills

Education

Bachelor's or Master's degrees