Description

Lead data engineering efforts on Databricks platforms, owning pipeline stability, performance optimization, and infrastructure management.

This role is on-site.

Responsibilities

• Monitor and troubleshoot PySpark-based data pipelines within Databricks environments.
• Debug and optimize ETL workflows for performance and reliability.
• Manage Databricks jobs, clusters, and workspace configurations.
• Collaborate with teams to resolve data issues and ensure pipeline stability.
• Integrate Azure services (Data Lake, Blob Storage) with DevOps for data operations.

Required Skills

• 5+ years of experience with PySpark and Databricks.
• Deep understanding of ETL workflow optimization and debugging.
• Experience managing Databricks clusters and workspace configurations.
• Proficiency with Azure Data Lake and Blob Storage.
• Familiarity with DevOps practices for data operations.
• Strong troubleshooting skills for distributed data systems.

Key Skills

Education

Any Graduate