Description

What You'll Do:

• Design, develop, and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build and optimize ETL/ELT workflows for processing structured, semi-structured, and unstructured datasets.
• Develop and manage Snowflake data warehouse solutions including data ingestion, transformation, and performance optimization.
• Create reusable frameworks for data ingestion, orchestration, transformation, validation, and monitoring.
• Integrate data from various enterprise applications, databases, APIs, and cloud platforms into Azure Data Lake and Snowflake environments.
• Design and implement modern data lake and Lakehouse architectures.
• Optimize data processing workloads for performance, reliability, scalability, and cost efficiency.
• Implement data quality, governance, security, and monitoring best practices.
• Collaborate with business stakeholders, solution architects, data analysts, and engineering teams to deliver data-driven solutions.
• Troubleshoot and resolve production issues related to data pipelines and cloud data platforms.
• Participate in code reviews, CI/CD implementations, deployment automation, and DevOps initiatives.
• Support data platform modernization efforts and drive continuous improvement initiatives.
• Contribute to architecture discussions and recommend best practices for scalable cloud data solutions.

Expertise You'll Bring:

• 6-10 years of experience in Data Engineering and Cloud Data Platform development.
• Strong hands-on expertise in Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Proven experience with Snowflake including data loading, data transformation, query optimization, data sharing, and performance tuning.
• Strong SQL expertise including complex query development, optimization, troubleshooting, and performance management.
• Experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), Azure Key Vault, and Azure DevOps.
• Good understanding of Data Warehousing principles, Dimensional Modeling, ETL/ELT frameworks, and Lakehouse architecture.
• Experience building and supporting enterprise-scale data pipelines and analytics platforms.
• Knowledge of data governance, data quality, metadata management, and security best practices.
• Experience with Git, CI/CD pipelines, source control management, and Agile methodologies.
• Strong understanding of distributed computing and cloud-native data architectures.
• Strong analytical, troubleshooting, debugging, and problem-solving skills.
• Excellent communication and stakeholder management abilities.
• Ability to work independently while collaborating effectively across distributed teams

Key Skills

Education

Any Graduate