Description

What You'll Do: 

• Design, develop, and maintain scalable ETL and ELT pipelines using Databricks, PySpark, SQL, and Notebooks.
• Build and manage data processing workflows across Bronze, Silver, and Gold layers using Delta Lake architecture.
• Develop batch and real-time streaming data pipelines using Spark Structured Streaming.
• Optimize Spark jobs, clusters, and workloads for performance, scalability, and cost efficiency.
• Implement and maintain data quality checks, validation frameworks, and monitoring processes.
• Collaborate with data architects, analysts, and business stakeholders to understand requirements and deliver robust data solutions.
• Develop and maintain reusable data engineering frameworks and components.
• Integrate Databricks solutions with cloud-native services on Azure, AWS, or GCP.
• Manage Databricks cluster configurations, autoscaling, job orchestration, and workload optimization.
• Implement CI/CD pipelines and code deployment processes using Git and DevOps best practices.
• Troubleshoot and resolve issues related to data pipelines, cluster performance, storage, and platform environments.
• Support production deployments and ensure platform reliability and operational stability.
• Contribute to architecture discussions and recommend improvements to data platform design and implementation.
• Mentor junior team members and share best practices in Databricks and Spark development.
• Drive continuous improvement initiatives across the data engineering ecosystem.

Expertise You'll Bring:

• 7 to 12 years of experience in Data Engineering, Big Data, Analytics, or Data Platform development.
• Hands-on experience delivering multiple enterprise-scale Databricks implementations and migration projects.
• Strong expertise in Databricks workspace administration, job orchestration, and notebook development.
• Extensive experience with Apache Spark, PySpark, Spark SQL, and distributed computing concepts.
• Deep understanding of Spark internals, execution plans, partitioning, caching, and optimization techniques.
• Strong experience implementing Delta Lake architecture and Medallion data design patterns.
• Hands-on experience building batch and streaming data processing solutions.
• Experience with cloud platforms such as Azure, AWS, or GCP, with strong expertise in at least one cloud ecosystem.
• Strong knowledge of Databricks cluster management, autoscaling, and resource optimization.
• Experience designing and implementing large-scale data lakes and modern data architectures.
• Proficiency in SQL development, query optimization, and performance tuning.
• Experience with Git, CI/CD pipelines, DevOps practices, and automated deployments.
• Strong understanding of data governance, security, access controls, and compliance standards.
• Experience working with structured, semi-structured, and unstructured datasets.
• Excellent analytical, debugging, troubleshooting, and problem-solving skills.
• Strong communication and stakeholder management abilities.
• Experience leading teams, mentoring engineers, and driving technical excellence

Key Skills

Education

Any Graduate