Description

Lead big data development and manage production support for the Data Management and Reporting system.

Monitor production jobs, debug failures, and resolve P1 system issues. Design and develop data loading strategies and shared components for cross-module use. Extract, stage, transform, and load data into Postgres SQL and Hive-based systems. Ingest data from Mainframe systems using Kafka and process large datasets with Apache Spark and Scala. Coordinate daily handshakes with offshore teams and provide on-call support between 5 PM and 9 PM EST, including weekend coverage.

Responsibilities

• Monitor production jobs and resolve P1 system issues.
• Design data loading strategies and shared components.
• Build ETL pipelines for Postgres SQL and Hive.
• Ingest Mainframe data via Kafka and process with Spark/Scala.
• Provide on-call support during EST evenings and weekends.

Required Skills

• 5+ years in big data environments.
• Proficiency in Scala and Apache Spark.
• Hands-on experience with Hadoop, Kafka, and Hive.
• Experience with Postgres SQL and distributed databases.
• Working knowledge of Airflow, Control M, or Oozie.
• Experience with Kubernetes or containerization tools.
• Strong architecture, design, and coding skills.
• Proven experience in production operations.

Preferred Skills

• Ability to work within Agile methodologies.

Key Skills

Education

Any Graduate