Description

Design and develop scalable data pipelines on Databricks using PySpark and Databricks SQL to deliver reliable data for analytics.

This role is on-site.

Responsibilities

• Build and optimize data transformation logic in Databricks Workflows to ensure data quality and business rule alignment.
• Configure Databricks CLI automation scripts to streamline environment setup, deployment, and operational tasks.
• Tune PySpark jobs and Databricks SQL queries to improve performance, manage costs, and optimize compute resources.
• Implement monitoring, logging, and alerting practices for Databricks Workflows to ensure reliability and rapid incident detection.
• Apply prompt engineering techniques to build intelligent data solutions that improve insight discovery and decision support.

Required Skills

• 6+ years of experience in data engineering and analytics solutions.
• Strong proficiency in Databricks and PySpark for scalable data processing.
• Experience with Databricks SQL for analytics and reporting needs.
• Ability to design and maintain scalable data pipelines and data architecture.
• Hands-on experience with Databricks CLI for automation and deployment.
• Skills in performance optimization and cost management for data platforms.
• Proficiency in Python and prompt engineering techniques.

Preferred Skills

• Experience with version control systems and dashboard development.
• Knowledge of security and privacy standards compliance for sensitive data.
• Mentoring capabilities to guide junior developers in Databricks and PySpark.

Key Skills

Education

Any Graduate