Description

Lead the design and optimization of large-scale ETL pipelines on Azure using PySpark and Databricks. You will own data ingestion, modeling, and governance while ensuring high-quality data delivery for analytics and science teams.

Responsibilities

• Design, develop, and optimize ETL pipelines using PySpark and Databricks on the Azure cloud platform.
• Implement data ingestion from diverse sources into Azure Data Lake and Azure SQL Data Warehouse.
• Develop and maintain data models, schemas, and transformation logic tailored for Azure environments.
• Collaborate with data scientists and analysts to define requirements and deliver high-quality datasets.
• Ensure data quality through rigorous testing, validation, monitoring, and performance tuning of PySpark jobs.

Required Skills

• 6+ years of experience as a Data Engineer with strong focus on PySpark and Databricks.
• Proficiency in Python and PySpark for large-scale data processing and analysis.
• Strong experience with Azure data services: Azure Data Lake, Azure Data Factory, Azure SQL Data Warehouse, and Azure Databricks.
• Strong SQL skills and experience with relational databases (MySQL, PostgreSQL) and NoSQL databases (MongoDB, Cassandra).
• Experience with big data technologies including Hadoop, Spark, Hive, and Kafka.
• Familiarity with Azure DevOps, Git, and CI/CD pipelines.
• Understanding of data architecture, data modeling, and data integration techniques.

Preferred Skills

• Experience with Delta Lake on Azure Databricks.
• Knowledge of data visualization tools (Power BI, Tableau) or containerization (Docker, Kubernetes).

Key Skills

Education

Bachelor's or Master's degrees