Description

You will build and maintain PySpark applications to automate data reconciliation and resolve discrepancies across financial systems.

Responsibilities

• Design and develop PySpark applications to automate reconciliation across relational databases, NoSQL databases, batch files, and real-time streams.
• Implement efficient data transformation and matching algorithms using deterministic and heuristic methods.
• Integrate PySpark applications with rules engines to execute complex data matching logic.
• Analyze data structures to identify gaps, inconsistencies, and integration challenges.
• Develop error handling and exception management mechanisms to maintain data integrity within Spark jobs.

Required Skills

• 5+ years of hands-on experience in big data development.
• Proficiency in PySpark and Apache Spark for large-scale data processing.
• Experience with NoSQL databases.
• Strong understanding of data reconciliation principles and techniques.
• Ability to translate business requirements into technical solutions for data matching.
• Bachelor's degree in Computer Science or a related field.

Preferred Skills

• Familiarity with Kafka or Kinesis.
• Experience with Hadoop, Hive, or HBase.

Key Skills

Education

Bachelor's degree in Computer Science