Description

Role Overview:

This role is for a Databricks Data Engineer with experience in designing, developing, and deploying robust, scalable batch and streaming data pipelines using PySpark, Spark SQL, and Delta Live Tables.

Key Responsibilities:

• Data Pipeline Development: Design, develop, and deploy data pipelines using PySpark, Spark SQL, and Delta Live Tables.
• Data Modeling & Transformation: Implement complex data transformations and business logic using the Medallion Architecture.
• Data Quality & Reliability: Design and implement data quality frameworks and cleansing routines.
• Performance & Cost Optimization: Proactively monitor, debug, and tune Databricks jobs and Spark clusters.
• Infrastructure as Code & CI/CD: Partner with DevOps teams to manage Databricks environments, clusters, and job deployments.
• Data Governance & Security: Implement governance capabilities using Databricks Unity Catalog.
• Collaboration: Work closely with Functional Consultants, Data Scientists, and Analytics Engineers.

Required Skills:

• Databricks & Spark Expertise: Deep hands-on experience with the Databricks Lakehouse Platform.
• Programming & Data Engineering: Expert-level proficiency in Python and PySpark.
• Data Warehousing Concepts: Strong knowledge of data modeling principles.
• Cloud & Software Engineering: Proven experience with a major cloud platform.

Preferred Qualifications:

• Databricks Certified Data Engineer – Professional certification.
• Experience with Terraform or other IaC tools.
• Experience delivering data solutions in retail or e-commerce environments

Key Skills

Education

Any Graduate