Description

You will design, build, and optimize scalable data pipelines using Databricks, PySpark, and Python on AWS.

This role is on-site.

Responsibilities

• Design and maintain ETL/ELT pipelines using Databricks, PySpark, and Python.
• Build and optimize large-scale data processing solutions on AWS.
• Implement data ingestion frameworks for structured and unstructured data.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost.
• Support CI/CD deployment and automation of data engineering workflows.

Required Skills

• 7+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks and Delta Lake.
• Expert programming skills in Python and PySpark.
• Deep knowledge of Apache Spark architecture and optimization techniques.
• Experience with AWS services: S3, Glue, Lambda, EMR, Redshift, IAM, CloudWatch, Secrets Manager.
• Strong SQL skills and experience with relational databases.
• Experience building batch and near real-time data processing solutions.
• Knowledge of Git, Jenkins, CI/CD pipelines, and Agile methodologies.

Preferred Skills

• Experience working with Unity Catalog and Databricks Workflows.
• Knowledge of banking data governance, security, and regulatory standards.

Key Skills

Education

Bachelor's degree