Description

You design, develop, and maintain scalable data pipelines and ETL/ELT processes using Azure Databricks and Apache Spark.

This role is on-site.

Responsibilities

• Build and optimize data processing frameworks using Java and Spark for large-scale structured and unstructured data.
• Write complex SQL queries, stored procedures, and performance-tuned solutions for enterprise data lakes and warehouses.
• Integrate data from multiple sources while implementing data quality, governance, and security standards.
• Monitor, troubleshoot, and optimize data processing jobs for performance and reliability.
• Support CI/CD deployment and automation of data engineering workflows.

Required Skills

• 10+ years of experience in Data Engineering.
• Proficiency with Azure Databricks and Apache Spark.
• Strong programming skills in Java for data transformation frameworks.
• Expert-level SQL, including complex queries and stored procedures.
• Experience designing ETL/ELT processes for large-scale data.
• Knowledge of CI/CD deployment and DevOps practices for data workflows.
• Understanding of data governance, security standards, and collaboration with data architects.

Preferred Skills

• Experience with data automation and workflow orchestration.
• Proven track record of optimizing data pipeline performance and reliability.

Key Skills

Education

Bachelor's degree