Description

Required Skills

• 5+ years of experience in Data Engineering.
• Strong hands-on experience with Databricks.
• Excellent programming skills in Python and PySpark.
• Strong knowledge of Apache Spark architecture and optimization techniques.
• Experience working with AWS services, including:
  • S3
  • Glue
  • Lambda
  • EMR
  • Redshift
  • IAM
  • CloudWatch
  • Secrets Manager
• Experience with Delta Lake and data lake architectures.
• Strong SQL skills and experience with relational databases.
• Experience building batch and near real-time data processing solutions.
• Knowledge of Git, Jenkins, CI/CD pipelines, and Agile methodologies.
• Strong analytical and problem-solving skills.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks, PySpark, and Python.
• Build and optimize large-scale data processing solutions on the AWS Cloud Platform.
• Develop and manage data ingestion frameworks for structured and unstructured data.
• Implement data transformation, cleansing, validation, and aggregation processes.
• Work closely with Data Architects, Business Analysts, and Banking stakeholders to understand data requirements.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
• Develop solutions using Delta Lake, Databricks Workflows, and Unity Catalog.
• Implement data quality checks, monitoring, and troubleshooting mechanisms.
• Support CI/CD deployment processes and automation of data engineering workflows.
• Ensure compliance with banking data governance, security, and regulatory standards.
• Participate in code reviews, performance tuning, and production support activities

Key Skills

Education

Bachelor's degree