Description

Job Description

• •Lead a team of Data Engineers and provide technical guidance and mentorship.
• •Design and implement scalable data solutions using Databricks and PySpark.
• •Define architecture, coding standards, and best practices, and oversee development, testing, deployment, and support activities.
• •Collaborate with business and technical stakeholders to gather requirements and define solutions.
• •Optimize data pipelines for performance, scalability, and reliability.
• •Manage sprint planning, task allocation, and delivery tracking, and support production environments to resolve critical issues.
• •Drive continuous improvement initiatives across the data platform.

Skills and Experience Required

• •Databricks and Apache Spark, with 4+ years hands on Databricks experience
• •Strong expertise in PySpark and advanced SQL with data modeling
• •Delta Lake, Unity Catalog, and Databricks Workflows
• •ETL and ELT pipeline design, data warehousing, and Lakehouse architecture
• •Proficiency in Python programming
• •Azure Data Factory, Azure Data Lake Storage, and similar cloud data services
• •RAG architectures, Azure OpenAI, OpenAI APIs, and GenAI frameworks such as LangChain or LlamaIndex
• •Git, CI/CD pipelines, and DevOps practices, plus data governance, security, and performance optimization
• •Databricks certification, with team leadership and Agile or Scrum delivery experience

Key Skills

Education

Any Graduate