Description

You will design and maintain large-scale data pipelines and ETL/ELT frameworks for the financial services sector.

Responsibilities

• Design ETL/ELT logical models and data structures for scalability and flexibility.
• Develop and implement data pipelines using Spark and Python.
• Build reusable components for ETL/ELT frameworks to streamline data integration.
• Construct and manage CI/CD pipelines across multiple integration and test environments.
• Integrate new data management technologies and software engineering tools into existing architectures.

Required Skills

• 10+ years of experience in the design, development, and implementation of large-scale projects.
• Expertise in PySpark and Spark for creating ETL transformations and jobs.
• Strong proficiency in Python and SQL.
• Hands-on experience with orchestration tools such as Airflow or Control-M.
• Experience with Big Data and distributed frameworks including Hadoop, Hive, and Kubernetes.
• Proven ability to design ETL/ELT solutions based on reporting and archival requirements.
• Experience building and managing CI/CD pipelines.
• Basic knowledge of Azure Cloud components.

Preferred Skills

• Degree in any field of study.

Key Skills

Education

Any Graduate