Description

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT workflows using Azure Data Factory (ADF), Azure Databricks, and Azure Synapse Analytics to process and transform large volumes of structured and unstructured data.
• Build and optimize big data processing solutions using Apache Spark, PySpark, Scala, or SQL within Azure Databricks, ensuring high performance and efficient data transformation.
• Design and implement modern data warehouse and lakehouse architectures using Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), and Azure SQL Database to support enterprise-level analytics and reporting.
• Develop and maintain data ingestion frameworks to integrate data from multiple sources such as databases, APIs, flat files, streaming platforms, and third-party applications into Azure-based data platforms.
• Implement data modeling techniques including star schema, snowflake schema, and dimensional modeling to support business intelligence and analytical workloads.
• Develop SQL-based data transformation and data quality validation processes using T-SQL, Spark SQL, and stored procedures to ensure accuracy, consistency, and reliability of data.
• Integrate data engineering solutions with business intelligence and analytics tools such as Power BI, Tableau, or Azure Analysis Services for reporting and data visualization.
• Implement real-time and streaming data processing solutions using Azure Event Hub, Azure Stream Analytics, or Apache Kafka to support near real-time analytics.
• Optimize data storage, partitioning, indexing, and query performance across Azure-based data platforms to ensure efficient data retrieval and processing.
• Implement data governance, security, and compliance using Azure Active Directory (AAD), Role-Based Access Control (RBAC), data encryption, and data masking techniques.
• Automate deployment and data pipeline orchestration using CI/CD pipelines with Azure DevOps, GitHub Actions, or Jenkins, ensuring reliable and consistent deployment processes.
• Monitor and troubleshoot data pipelines, workflows, and cloud resources using Azure Monitor, Log Analytics, and Application Insights to ensure system reliability and performance.
• Collaborate with data architects, DevOps engineers, and cross-functional Agile teams to design scalable data engineering solutions aligned with enterprise data strategies.
• Mentor junior data engineers and contribute to best practices in data engineering, cloud architecture, and performance optimization.

Qualifications

10+ Years of experience in data engineering.

Required Skills

• Cloud Platform: Microsoft Azure
• Azure Data Services: Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage (ADLS Gen2), Azure SQL Database
• Big Data Technologies: Apache Spark, PySpark, Scala, Spark SQL
• Programming Languages: Python, SQL, Scala
• Data Warehousing: Azure Synapse Analytics, Azure SQL Data Warehouse, Snowflake Concepts, Dimensional Modeling
• ETL/ELT Tools: Azure Data Factory, Databricks Workflows
• Streaming Technologies: Azure Event Hub, Azure Stream Analytics, Apache Kafka
• Databases: SQL Server, Azure SQL Database, PostgreSQL, MySQL, NoSQL
• Data Visualization Tools: Power BI, Tableau
• Version Control: Git, GitHub, Bitbucket, GitLab
• CI/CD Tools: Azure DevOps, Jenkins, GitHub Actions
• Containerization: Docker, Kubernetes
• Operating Systems: Windows, Linux
• Methodologies: Agile, Scrum, DevOps, DataOps

Preferred Skills

Experience with data governance and compliance frameworks

Key Skills

Education

Bachelor's degree