Description

You will build and optimize data pipelines using the Databricks Unified Analytics Platform.

Responsibilities

• Design and implement ETL processes using Spark SQL, DataFrame API, and structured streaming.
• Configure and optimize Databricks clusters, runtime settings, and autoscaling for efficient resource utilization.
• Automate tasks within the platform using APIs and the Databricks CLI.
• Monitor workloads, interpret logs, and optimize performance using monitoring tools.
• Manage codebase changes and collaborate with the team using Git and Databricks notebooks.

Required Skills

• 5+ years of experience in data engineering roles.
• Proficiency with the Databricks Unified Analytics Platform, including notebooks, clusters, jobs, and libraries.
• Strong programming skills in Python, Scala, and SQL.
• Deep understanding of Spark SQL, DataFrame API, and structured streaming.
• Mastery of Spark DataFrames and Datasets for structured data manipulation.
• Experience connecting Databricks to cloud storage, databases, and streaming platforms.
• Knowledge of Databricks security features, access controls, and encryption.
• Experience with version control systems like Git.

Preferred Skills

• Degree in any field (Any Graduate).

Key Skills

Education

Any Graduate