Databricks Technical Lead

You will lead the design, build, and optimization of a large-scale data platform, services, and APIs within an agile framework.

Responsibilities

• Lead a data engineering team in designing scalable, high-performance data solutions, APIs, and microservices.
• Design and implement scalable data pipelines and architectures using Azure Databricks and Apache Spark.
• Optimize ETL/ELT workflows and fine-tune Spark jobs for performance and cost efficiency.
• Manage the migration of Azure SQL workloads to Azure Databricks.
• Maintain the HR data domain ensuring strict adherence to security, privacy, and compliance standards.

Required Skills

• 10+ years of experience in data engineering, big data, or API development.
• 3+ years in a technical leadership role.
• Expertise in Azure Databricks, Apache Spark, Azure SQL, and Delta Lake.
• Proficiency in Python, Scala, and SQL.
• Experience building and managing REST, GraphQL, or gRPC APIs and microservices.
• Hands-on experience with Azure Data Factory (ADF) and Azure Synapse Analytics.
• Proficiency in CI/CD pipelines, Terraform/Bicep, and Infrastructure-as-Code.
• Experience with data security measures including encryption, access control, and auditing.
• Knowledge of Informatica for ETL/ELT and data integration.

Preferred Skills

• Containerization experience with Docker and Kubernetes.
• Familiarity with event-driven architecture and Azure Event Hub.
• Experience with Oracle HCM database models and HR Analytics.