Description

You will design and build enterprise-grade data solutions across the data lifecycle.

Responsibilities

• Design, build, and optimize ETL/ELT pipelines for ingestion, transformation, and publishing using ADF, Spark, and Databricks.
• Design and implement end-to-end data architecture for batch and real-time processing using Azure and Microsoft Fabric.
• Develop and manage scalable data processing frameworks using Databricks and Apache Spark.
• Develop efficient data transformation logic using PySpark and Spark SQL.
• Define and enforce data engineering best practices, including CI/CD, versioning, testing, and monitoring.

Required Skills

• 12+ years of experience in data architecture and engineering.
• Strong hands-on experience with Azure Data Factory (ADF) for data orchestration.
• Proficiency in PySpark, Python, and Spark SQL.
• Experience designing data lakes, data warehouses, or Lakehouse architectures.
• Expertise in SQL and relational data modeling.
• Experience with CI/CD pipelines for data platforms, including Azure DevOps and GitHub Actions.
• Familiarity with data warehousing technologies such as Azure Synapse and Azure SQL.
• Solid understanding of distributed data processing and performance tuning.
• Experience with ETL/ELT patterns and best practices.

Key Skills

Education

Bachelor's degree