Description

Mandatory skills:

§ Strong hands-on experience with Databricks, PySpark, and SQL.

§ Experience with Azure, AWS, or GCP cloud platforms.

§ Expertise in Delta Lake, Delta Live Tables (DLT), Unity Catalog/Data Governance, Databricks Workflows, and Serverless Compute.

§ Knowledge of Genie, Mosaic AI, and Databricks Data Intelligence Platform capabilities.

§ Strong understanding of Data Architecture, Data Modelling, Lakehouse Architecture, Data Warehousing, and Medallion Framework.

§ Experience with GitHub, CI/CD pipelines, Agile delivery models, and DevOps practices.

§ Strong analytical, troubleshooting and performance optimization skills.

§ Excellent stakeholder management and business requirement gathering capabilities.

§ Ownership to bring initiatives on performance & cost optimization as required.

Responsibilities

§ Develop and maintain scalable data engineering solutions using Databricks, PySpark, and SQL for enterprise data platforms.

§ Design, implement, and optimize data architectures that are secure, scalable, reliable, and performance-driven.

§ Build and manage data pipelines for processing structured, semi-structured, and unstructured data from multiple source systems.

§ Develop and support both batch and near real-time data processing solutions using Databricks and Spark technologies.

§ Implement and manage Databricks capabilities including Delta Lake, Delta Live Tables (DLT), Data Workflows, Serverless Compute, Unity Catalog/Data Governance, Genie, Mosaic AI, and the Databricks Data Intelligence Platform.

§ Design and develop data solutions following Lakehouse, Medallion (Bronze-Silver-Gold), Data Warehouse, and Data Mart architectural standards.

§ Optimize Spark jobs, SQL queries, clusters, and data pipelines to improve performance, scalability, reliability, and cost efficiency.

§ Manage and monitor Databricks clusters, resource utilization, and workload performance across cloud environments.

§ Collaborate with business stakeholders, product owners, and architects to understand business requirements and translate them into scalable technical solutions.

§ Define and implement enterprise data governance, security, access controls, lineage, and compliance standards within Databricks.

§ Leverage cloud-native services across Azure, AWS, or GCP to build integrated and scalable data solutions.

§ Develop and deploy code using GitHub, CI/CD pipelines, and Agile development methodologies, ensuring adherence to coding standards and best practices.

§ Work with large-scale datasets while implementing robust security models, data quality frameworks, and performance optimization techniques.

§ Perform system integration and end-to-end testing to ensure data accuracy, completeness, and reliability across platforms.

§ Analyze, troubleshoot, and resolve defects, data issues, and production incidents in a timely manner, ensuring minimal business impact.

§ Facilitate discussions with business and technical teams to proactively address data, integration, and platform-related challenges.

§ Ensure effective management of resources, dependencies, and deliverables related to data engineering and integration initiatives.

§ Document technical designs, data models, architecture decisions, and operational procedures to support long-term platform sustainability.  

Qualifications

§ Bachelor’s or master’s degree in any field

§ Your experience in designing, implementing, and maintaining data solutions on Databricks

§ Experience with at least one of the popular cloud platforms – Azure, AWS or GCP

§ Experience with ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) processes

§ Knowledge of data warehousing and data modelling concepts

§ Experience with Python or SQL

§ Experience with Delta Lake

§ Understanding of DevOps principles and practices

§ Excellent problem-solving and troubleshooting skills

§ Strong communication and teamwork skills

Key Skills

Education

Any Graduate