Description

Key Skills: Databricks, PySpark, Databricks SQL, Data Pipelines, Data Engineering, Prompt Engineering, Databricks CLI, Performance Optimization, Data Architecture, Python

Good to Have Skills: Experience with hybrid work environments, monitoring and logging practices, version control systems, dashboard development, security and privacy standards compliance, mentoring capabilities, and innovative use case development for societal impact.

Roles & Responsibilities:

• Design and develop scalable data pipelines on Databricks using PySpark and Databricks SQL to deliver reliable and timely data for analytics and reporting needs.
• Implement robust data transformation logic in Databricks Workflows to ensure consistent data quality and alignment with business rules across multiple domains.
• Configure and optimize Databricks CLI based automation scripts to streamline environment setup deployment and operational tasks for data engineering projects.
• Apply prompt engineering techniques to build and refine intelligent data solutions that improve insight discovery and decision support for business teams.
• Tune PySpark jobs and Databricks SQL queries to improve performance manage costs and ensure efficient use of compute resources across the data platform.
• Collaborate with data architects analysts and product teams in a hybrid work setup to translate complex requirements into clear maintainable Databricks based solutions.
• Establish monitoring logging and alerting practices for Databricks Workflows to ensure high reliability rapid incident detection and swift remediation.
• Document data models transformation logic and operational runbooks so that project teams can understand support and extend delivered solutions with confidence.
• Ensure all data solutions comply with security and privacy standards so that sensitive information is protected while enabling responsible insight generation.
• Mentor junior developers in Databricks PySpark and prompt engineering techniques so that the team continually improves its technical depth and delivery quality.
• Partner with business stakeholders to validate outputs from Databricks SQL dashboards and models so that delivered insights remain accurate relevant and actionable.
• Drive continuous improvement of development practices using Databricks CLI and version control so that releases remain predictable traceable and low risk.
• Contribute to innovative use cases that leverage Databricks and prompt engineering to generate positive business value and broader societal impact through better use of data.

Experience Required: 6 to 9 years of experience in data engineering and analytics solutions

Key Skills

Education

Any Graduate