Description

Design and govern enterprise data platforms leveraging Hadoop and PySpark for surveillance and regulatory workloads.

This role is hybrid.

Responsibilities

• Define target state architecture for surveillance data processing using Hadoop, Spark, and Python.
• Design scalable ELT/ETL pipelines, data lakes, and real-time batch processing solutions.
• Translate business requirements into technical designs for alert generation, enrichment, and reporting.
• Modernize large-scale Hadoop workloads to improve performance, stability, and audit readiness.
• Mentor senior engineers and lead architecture review forums to ensure governance and best practices.

Required Skills

• 10+ years of experience in Data Engineering and Platform Engineering.
• Deep expertise in PySpark with performance tuning and optimization.
• Strong Python development experience in large-scale distributed environments.
• Solid knowledge of Hadoop ecosystem components: HDFS, Hive/Impala, and YARN.
• Proficiency in Java, Scala, and SQL.
• Proven experience designing enterprise data platforms with a focus on governance and lineage.
• Understanding of non-functional requirements, production support models, and data quality controls.

Preferred Skills

• Financial services experience, particularly in Market Surveillance, AML, or Fraud.
• Experience with Kafka, Spark Structured Streaming, and orchestration tools like Airflow or Control-M.

Key Skills

Education

Bachelor's or Master's degrees