Description

Design and maintain scalable data pipelines using Databricks, PySpark, and Python on AWS.

This role is on-site.

Responsibilities

• Build and optimize large-scale ETL/ELT pipelines and data ingestion frameworks for structured and unstructured data.
• Implement data transformation, cleansing, validation, and aggregation using Delta Lake, Databricks Workflows, and Unity Catalog.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
• Support CI/CD deployment processes and automation of data engineering workflows.
• Implement data quality checks, monitoring, and troubleshooting mechanisms while ensuring compliance with banking data governance standards.

Required Skills

• 7+ years of experience in data engineering.
• Proficiency in Python and PySpark.
• Hands-on experience with Databricks and Delta Lake.
• Strong background in AWS Cloud Platform.
• Experience with Spark optimization and workload tuning.
• Familiarity with Jenkins and CI/CD pipelines.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience collaborating with Data Architects and Business Analysts in banking environments.
• Knowledge of data governance, security, and regulatory compliance standards.

Key Skills

Education

Bachelor's degree