Description

You will design, build, and maintain large-scale data infrastructure and ETL pipelines for financial services.

This role is hybrid.

Responsibilities

• Design and implement scalable ETL pipelines using PySpark, SQL, and Delta Lake on Databricks.
• Tune Spark jobs to handle datasets in the billions of records, utilizing partitioning and caching strategies.
• Develop and optimize calculation engines that integrate with multiple source systems.
• Implement parameterized financial logic, including APR and compounding rules, based on business requirements.
• Ensure data quality, lineage, and reconciliation across all data layers.

Required Skills

• 15+ years of experience in Data Engineering, preferably within financial services.
• Strong hands-on experience with PySpark and SQL.
• Expertise with Databricks, including Delta Live Tables and Unity Catalog.
• Solid understanding of data modeling, partitioning strategies, and query optimization.
• Experience implementing CI/CD using tools like GitHub or Jenkins.
• Familiarity with workflow orchestrators such as Airflow or Dagster.
• Experience handling credit card or banking domain data.

Preferred Skills

• Deep expertise in financial data logic and reconciliation.
• Experience with large-scale distributed systems optimization.

Key Skills

Education

Any Graduate