Description

You will build and maintain data pipelines to ingest, process, and analyze large-scale datasets. You will own the full lifecycle of batch and real-time processing systems while ensuring data integrity for downstream analytics and machine learning models.

Responsibilities

• Develop and maintain data pipelines for batch and real-time processing.
• Cleanse, validate, and process data to ensure integrity for downstream analysis.
• Collaborate with data science, business, and IT teams to implement analytics pipelines and machine learning algorithms.
• Define data governance policies and support data-versioning and security processes.
• Design, test, and implement system applications including technical and functional specifications.

Required Skills

• 6+ years of software development experience with a track record of leading engineering and scrum teams.
• 5+ years of hands-on experience with Informatica and Oracle.
• Expertise in the Hadoop and Spark ecosystems, including architecture and working principles.
• Proficiency in Core Java, Python, or Scala.
• Advanced SQL skills for writing complex Hive or PySpark data frames and optimizing joins on large datasets.
• Experience with UNIX shell scripting.
• Strong understanding of data warehousing concepts and distributed ecosystems.
• Bachelor's Degree in Engineering, Computer Science, or equivalent.

Preferred Skills

• 5+ years of experience in Data Engineering or Data Warehousing using Big Data technologies.
• Knowledge of the Financial reporting ecosystem.

Key Skills

Education

Bachelor's Degree