Description

Design and govern enterprise data platforms and surveillance architectures using Hadoop, Spark, and Python.

This role is hybrid.

Responsibilities

• Define target state architecture for surveillance data processing platforms leveraging Hadoop, Spark, Pyspark, and Python.
• Translate business requirements into scalable technical designs for alert generation, enrichment, and reporting pipelines.
• Modernize large-scale Hadoop workloads to improve performance, stability, and regulatory traceability.
• Lead architecture review forums, mentor senior engineers, and promote adoption of standard frameworks.
• Maintain high-quality documentation for business requirements, data definitions, and process flows.

Required Skills

• 10+ years of experience in large-scale distributed environments.
• Deep expertise in PySpark, including performance tuning and optimization.
• Strong Python development experience in enterprise settings.
• Solid knowledge of the Hadoop ecosystem: HDFS, Hive/Impala, and YARN.
• Proficiency in Java, Scala, and SQL.
• Proven experience designing data lakes, ELT/ETL pipelines, and batch/real-time processing solutions.
• Understanding of non-functional requirements and production support models.
• Bachelor's or Master's degree in a relevant field.

Preferred Skills

• Financial services experience, particularly in Market Surveillance, AML, Fraud, or Risk Technology.
• Experience supporting regulatory or audit-facing platforms.
• Familiarity with Kafka, Spark Structured Streaming, and orchestration tools like Airflow, Control-M, or Oozie.

Key Skills

Education

Bachelor's or Master's degrees