Description

You will design and build scalable data pipelines and workflows using Databricks across cloud environments.

Responsibilities

• Architect and build scalable data pipelines and workflows using Databricks on AWS, Azure, or GCP.
• Optimize, transform, and cleanse large datasets to enable data-driven insights.
• Fine-tune Databricks workflows for performance, scalability, and cost-effectiveness.
• Develop reusable frameworks for data processing and implement CI/CD workflows.
• Integrate various data sources into unified solutions by collaborating with data scientists and analysts.

Required Skills

• 5+ years of experience with Databricks and Spark technologies.
• Proficiency in Python, Scala, or SQL.
• Hands-on experience with AWS, Azure, or GCP.
• Experience with Hadoop and Hive.
• Ability to translate business requirements into technical solutions.
• Experience maintaining technical documentation for pipelines and models.

Education

• Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.

Key Skills

Education

Bachelor’s or Master’s degree