Description

You will own the design, security, and automation of data infrastructure within the Databricks ecosystem.

Responsibilities

• Manage and secure data assets using Databricks Unity Catalog.
• Build and maintain monitoring and observability frameworks using Datadog.
• Develop Python and PySpark scripts for data processing and analysis.
• Configure CI/CD pipelines using GitHub Actions for workflow automation.
• Apply version control best practices with GitHub and leverage AI coding assistants.

Required Skills

• 5+ years of experience with Databricks, Spark, and Delta Lake.
• Proficiency in Python and PySpark for data engineering tasks.
• Hands-on experience with Unity Catalog for data governance and security.
• Strong understanding of data architecture and dataset structures.
• Experience with Azure cloud services and PowerBI for visualization.
• Practical knowledge of Mlflow for machine learning lifecycle management.
• Ability to interpret and understand complex dataset requirements.

Preferred Skills

• Familiarity with Databricks Assistant and GitHub Copilot for development efficiency.
• Experience implementing DevOps principles in data engineering workflows.

Key Skills

Education

Any Gradute