Description

You will build and maintain data pipelines to process large-scale data within the Azure ecosystem.

Responsibilities

• Design, develop, and optimize ETL pipelines using PySpark and Databricks on Azure.
• Implement data ingestion processes from diverse sources into Azure Data Lake and Azure SQL Data Warehouse.
• Develop and maintain data models, schemas, and transformation logic specific to Azure requirements.
• Collaborate with data scientists and analysts to deliver required, high-quality datasets.
• Ensure data quality and integrity through rigorous testing, validation, and monitoring.

Required Skills

• 6+ years of proven experience as a Data Engineer focusing on PySpark and Databricks.
• Proficiency in Python and PySpark for data processing and analysis.
• Strong experience with Azure data services: Azure Data Lake, Azure Data Factory, Azure SQL Data Warehouse, and Azure Databricks.
• Strong SQL skills covering relational (MySQL, PostgreSQL) and NoSQL (MongoDB, Cassandra) databases.
• Experience with big data technologies including Hadoop, Spark, Hive, and Kafka.
• Strong understanding of data architecture, modeling, and integration techniques.
• Familiarity with Azure DevOps, Git, and CI/CD pipelines.
• Knowledge of data integration formats like JSON, CSV, XML, Parquet, and Delta.

Key Skills

Education

Bachelor's or Master's degrees