Description

Lead the architecture and design of Databricks-based solutions for data engineering, machine learning, and real-time analytics.

Responsibilities

• Design and implement ETL pipelines using Databricks, Apache Spark, and big data tools to integrate large-scale data.
• Create scalable, cost-effective architectures for processing large datasets using Delta Lake and Apache Spark.
• Manage and optimize Databricks clusters for performance, reliability, and cloud resource efficiency.
• Implement data governance, security, and compliance standards on the Databricks platform.
• Mentor junior engineers and define best practices for data pipeline development and monitoring.

Required Skills

• 5+ years of experience in data architecture and engineering with cloud platforms (AWS, Azure, or GCP).
• Extensive experience with Databricks and Apache Spark.
• Proficiency in Python, Scala, or SQL.
• Hands-on experience with Delta Lake, Spark SQL, and MLlib.
• Expertise deploying workloads on AWS, Azure, or GCP.
• Knowledge of cloud-native services such as S3, Redshift, Azure Blob Storage, or BigQuery.
• Experience designing, building, and optimizing ETL pipelines.
• Understanding of data modeling, distributed computing, and OLAP/OLTP systems.
• Ability to integrate machine learning workflows and automate model deployment.

Preferred Skills

• Experience with DevOps, automation, and CI/CD pipelines in data engineering.
• Databricks Certified Associate Developer for Apache Spark.
• Cloud certifications such as AWS Certified Solutions Architect or Azure Solutions Architect Expert.

Key Skills

Education

Bachelor's Degree