Description

Key Responsibilities

• Lead the design, development, and maintenance of enterprise-scale data platforms on AWS and Databricks.
• Architect and implement scalable ETL/ELT pipelines using PySpark, Spark SQL, and Databricks Workflows.
• Build and optimize Delta Lake architectures, ensuring data quality, reliability, and performance.
• Design and implement data ingestion pipelines for structured, semi-structured, and streaming data.
• Collaborate with business stakeholders, architects, analysts, and data scientists to gather requirements and deliver data solutions.
• Optimize Spark jobs, SQL queries, and cluster configurations for performance and cost efficiency.
• Implement data governance, security, and access controls using Unity Catalog.
• Develop reusable frameworks, coding standards, and best practices for data engineering.
• Lead code reviews, mentor junior engineers, and provide technical leadership.
• Build CI/CD pipelines and automate deployment processes.
• Monitor production jobs, troubleshoot failures, and ensure SLA compliance.
• Support cloud migration and modernization initiatives from legacy systems to AWS Databricks.

Required Skills

• 10+ years of experience in Data Engineering.
• 4+ years of hands-on experience with Databricks.
• Strong experience with AWS Cloud services including:
  • S3
  • EMR
  • Glue
  • Lambda
  • IAM
  • CloudWatch
  • Redshift
  • RDS
  • EC2
• Expertise in PySpark, Spark SQL, and Python.
• Strong SQL development and query optimization skills.
• Experience with Delta Lake, Delta Live Tables (DLT), and Unity Catalog.
• Experience building batch and streaming data pipelines.
• Knowledge of Apache Spark architecture and performance tuning.
• Experience with Git, CI/CD, and DevOps practices.
• Familiarity with Agile/Scrum methodologies

Key Skills

Education

Bachelor's degree