Description

You will build, improve, and validate reliability patterns for a Databricks-based data platform during development.

Responsibilities

• Develop, maintain, and optimize Databricks notebooks, jobs, and workflows.
• Write and enhance PySpark and Python code for efficient data processing.
• Design scalable and reliable data pipelines, applying Spark best practices.
• Perform data validation and quality checks during development cycles.
• Collaborate with Data Engineers to prepare solutions for production deployment and document standards.

Required Skills

• 3+ years hands-on experience in Databricks and Spark ecosystems.
• Strong knowledge of PySpark and Spark fundamentals, including execution concepts (jobs, stages, tasks).
• Proficiency in Python development, capable of writing clean, modular, and reusable code.
• Experience working with Delta Lake tables and data models.
• Familiarity with Git and version control for development workflows.
• Demonstrated ability to identify performance bottlenecks and recommend improvements.
• Experience with Azure Cloud services is required.
• Bachelor's degree required.

Preferred Skills

• Experience with Microsoft Azure, specifically Azure Databricks, ADLS, or Blob Storage.
• Familiarity with CI/CD pipelines for data platforms.

Key Skills

Education

Any Graduate