Description

Own the architecture and development of large-scale Hadoop-based surveillance data platforms, ensuring regulatory compliance and performance.

This role is hybrid.

Responsibilities

• Define target state architecture for surveillance data processing using Hadoop, Spark, PySpark, and Python.
• Design and govern enterprise data lakes, ELT/ETL pipelines, and batch/real-time processing solutions.
• Translate business requirements into scalable technical designs for alert generation, enrichment, and reporting.
• Lead architecture review forums, mentor senior engineers, and enforce standard architectural patterns.
• Optimize large-scale Hadoop workloads for performance, stability, and audit readiness.

Required Skills

• 10+ years of experience in Data or Platform Engineering.
• Deep expertise in PySpark, including performance tuning and optimization.
• Strong Python development experience in large-scale distributed environments.
• Solid knowledge of the Hadoop ecosystem: HDFS, Hive/Impala, YARN.
• Proficiency in Java, Scala, and SQL.
• Experience designing data lakes and governing regulatory-facing data platforms.
• Bachelor's or Master's degree in a relevant field.

Preferred Skills

• Financial services experience, particularly in Market Surveillance, AML, Fraud, or Risk Technology.
• Exposure to Kafka, Spark Structured Streaming, and orchestration tools like Airflow or Control-M.
• Familiarity with data governance, lineage, and data quality controls.

Key Skills

Education

Bachelor's or Master's degrees