Description

Job Description:-

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
• Build and optimize ETL/ELT workflows for processing structured and semi-structured data.
• Develop and manage Delta Lake tables and Lakehouse architecture.
• Create and maintain Databricks notebooks, jobs, and workflows.
• Optimize Spark applications for performance, scalability, and cost efficiency.
• Ensure data quality, governance, security, and compliance standards.
• Integrate data from multiple sources including databases, APIs, and cloud storage.
• Collaborate with Data Engineers, Data Scientists, Business Analysts, and stakeholders to deliver data solutions.
• Implement CI/CD practices and automate deployment processes.
• Monitor, troubleshoot, and resolve data pipeline and platform issues.

Required Skills

• Strong experience with Databricks, Apache Spark, and PySpark.
• Proficiency in SQL and Python.
• Hands-on experience with Delta Lake and Lakehouse architecture.
• Experience building ETL/ELT pipelines and data transformation workflows.
• Knowledge of cloud platforms such as Azure, AWS, or GCP.
• Understanding of Data Warehousing and Data Modeling concepts.
• Experience with Git, CI/CD, and DevOps practices.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and collaboration abilities.

Preferred Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.
• Databricks Certification (Associate/Professional) preferred.
• Experience with Azure Data Factory (ADF), Synapse, Kafka, or Airflow.
• Knowledge of Unity Catalog, DataOps, and data governance frameworks.

Nice to Have

• Experience with real-time streaming using Spark Streaming or Kafka.
• Knowledge of Terraform, Azure DevOps, or cloud infrastructure automation.
• Exposure to machine learning workflows on Databricks.

Keywords: Databricks, PySpark, Spark SQL, Delta Lake, ETL, ELT, Data Engineering, Azure Databricks, AWS Databricks, Lakehouse, Data Pipeline, Python, SQL, Data Warehousing

Key Skills

Education

Any Graduate