Description

Lead data engineering initiatives by designing and developing scalable data pipelines on Databricks and AWS.

This role is on-site.

Responsibilities

• Design and develop robust data pipelines using Databricks on AWS infrastructure.
• Collaborate with Product, Business, and UX teams to prioritize features and own project sizing and estimation.
• Troubleshoot and tune performance across integrated systems and production environments.
• Guide the Production Support team through complex technical issues and system optimizations.

Required Skills

• 10+ years of experience in data engineering.
• Strong command of SQL and proficiency in PySpark.
• Deep knowledge of Databricks architecture, tools, and capabilities.
• Experience designing and optimizing ETL/ELT pipelines using Databricks.
• Experience with Delta Lake and building data lakes.
• Proficiency with AWS services including S3, DataSync, and Lambda.
• Familiarity with CI/CD pipelines using Gitlab.
• Experience using Jira for project tracking.
• Knowledge of Spark internals and optimization.

Key Skills

Education

Any Gradute