Description

Key Responsibilities:

• Design, develop, and maintain ETL/ELT pipelines using Databricks and PySpark
• Build scalable batch and real-time data processing solutions using Apache Spark
• Develop data transformation workflows using Python, PySpark, and Spark SQL
• Implement Delta Lake solutions and optimize data processing performance
• Create and manage Databricks notebooks, jobs, and workflows
• Ensure data quality, validation, logging, and exception handling
• Optimize Spark performance using partitioning, caching, and tuning techniques
• Collaborate with cross-functional teams to gather requirements and deliver solutions
• Support CI/CD, deployment automation, and production releases
• Troubleshoot and resolve issues in data pipelines and platform processes

Required Skills:

• Strong hands-on experience with Databricks
• Strong Python programming skills
• Strong in communication and proactive.
• Expertise in Apache Spark, PySpark, and Spark SQL- Strong SQL and data transformation skills
• Experience with Delta Lake and lakehouse concepts- Knowledge of ETL/ELT and data pipeline design
• Experience with AWS, Azure, or GCP- Familiarity with Git, CI/CD, and Agile practices
• Good problem-solving and communication skills

Preferred Skills:

• Experience with Airflow, Azure Data Factory, or Databricks Workflows
• Knowledge of Kafka or streaming frameworks
• Experience with MLflow or analytics platforms
• Understanding of data governance and security best practices

Key Skills

Education

Bachelor's degree