Description

You will design and implement scalable data solutions using Databricks to support advanced analytics and data processing.

Responsibilities

• Develop and maintain ETL processes and data pipelines on the Databricks platform.
• Design and optimize scalable data models and schemas for data lakes and warehouses.
• Implement data ingestion, cleansing, and transformation tasks to ensure data integrity.
• Collaborate with data scientists to enable machine learning capabilities using Databricks notebooks and MLflow.
• Monitor performance, troubleshoot processing issues, and optimize queries for efficient storage and compute.

Required Skills

• 5+ years of experience in data engineering or a similar role.
• Strong programming proficiency in Python and SQL.
• Hands-on experience with Azure Databricks.
• Experience building data pipelines and ETL processes.
• Proficiency in data modeling and schema design for data lakes and warehouses.
• Working knowledge of cloud platforms including AWS, Azure, or GCP.
• Experience with cloud storage services such as S3, Blob Storage, or Data Lake Storage.
• Familiarity with MLflow and Databricks notebooks.

Preferred Skills

• Experience with Sqoop.
• Familiarity with Agile methodologies and Git version control.

Key Skills

Education

Any Graduate