Description

Build and optimize scalable data pipelines on Databricks and AWS.

This role is on-site.

Responsibilities

• Design and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and Python.
• Build and optimize large-scale data processing solutions on AWS.
• Develop data ingestion frameworks for structured and unstructured data.
• Implement data transformation, cleansing, and validation processes.
• Optimize Spark jobs and Databricks workloads for performance and cost.

Required Skills

• 7+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Excellent programming skills in Python and PySpark.
• Deep knowledge of Apache Spark architecture and optimization.
• Experience with AWS services: S3, Glue, Lambda, EMR, Redshift, IAM, CloudWatch, Secrets Manager.
• Experience with Delta Lake and data lake architectures.
• Strong SQL skills and experience with relational databases.
• Knowledge of Git, Jenkins, CI/CD pipelines, and Agile methodologies.

Preferred Skills

• Experience with banking data governance and regulatory standards.
• Knowledge of Unity Catalog and Databricks Workflows.

Key Skills

Education

Bachelor's degree