Description

You will design and build big data systems using PySpark, Python, and Scala, focusing on high-performance batch and real-time processing. You will refactor legacy, undocumented code into efficient, maintainable solutions, optimizing compute costs and data workflows. Your work involves managing large datasets, implementing machine learning models, and ensuring data integrity within warehousing environments.

Responsibilities

• Develop and maintain scalable data pipelines using PySpark, Hadoop, and Hive for batch and real-time processing.
• Refactor unstructured or legacy codebases into optimized, high-performance engineering standards.
• Apply supervised and unsupervised machine learning techniques for data preprocessing and application engineering.
• Design software systems with intense computational needs, handling large-scale datasets and data warehouses.
• Collaborate with stakeholders to articulate complex business problems and deliver structured technical solutions.

Required Skills

• Strong proficiency in Python and Scala programming languages.
• Hands-on experience with PySpark, Hadoop, Hive, and SQL.
• Experience with Unix scripting and data warehousing concepts.
• Ability to work with unstructured data and re-engineer costly compute processes.
• Understanding of machine learning techniques and data preprocessing.
• Bachelor's degree or equivalent experience with 5+ years in big data systems.

Preferred Skills

• Experience in Credit Cards and Retail Banking domains.
• Familiarity with Jenkins, AutoSys, and Service Now.

Key Skills

Education

Any Graduate