Description

You design, build, and maintain scalable data pipelines and analytics solutions using Databricks and Apache Spark. This role is remote.

Responsibilities

• Design and implement batch and streaming ETL pipelines using Databricks (PySpark/Spark SQL).
• Integrate data from diverse banking systems, including credit bureaus, transaction systems, and CRM.
• Ensure data quality, integrity, and lineage, implementing Delta Lake features for ACID compliance.
• Optimize Spark jobs for performance and manage job scheduling to meet SLAs.
• Enforce data governance and security policies to ensure compliance with banking regulations (AML, BSA, GDPR).

Required Skills

• 10+ years of professional experience in data engineering.
• Expertise in PySpark and Spark SQL.
• Experience with Azure and AWS cloud platforms.
• Familiarity with Kafka and Hadoop ecosystems.
• Proficiency in building data pipelines for fraud detection and risk modeling.
• Ability to support ML model deployment workflows.
• Experience with MLflow.

Key Skills

Education

Any Graduate