Description

You will design, develop, and maintain scalable data pipelines using Azure Databricks and Apache Spark to support large-scale structured and unstructured data processing.

Responsibilities

• Build and optimize ETL/ELT processes and data transformation frameworks using Java and Spark.
• Write complex SQL queries, stored procedures, and performance-tuned data solutions for enterprise data lakes and warehouses.
• Integrate data from multiple sources while implementing data quality, governance, and security standards.
• Monitor, troubleshoot, and optimize data processing jobs for performance and reliability.
• Support CI/CD deployment and automation of data engineering workflows in collaboration with cross-functional teams.

Required Skills

• 5+ years of experience in Data Engineering with Azure Databricks.
• Strong proficiency in Apache Spark and Java for data transformation and processing frameworks.
• Expertise in ETL/ELT processes and data pipeline design for large-scale data.
• Advanced SQL skills, including complex queries and stored procedures.
• Experience with data integration into enterprise data lakes and warehouses.
• Knowledge of data governance, security standards, and data quality implementation.
• Familiarity with CI/CD deployment and automation of data engineering workflows.
• Bachelor's degree in a relevant field.

Key Skills

Education

Bachelor's degree