Description

Experience: 10-12+ Years 

Primary Skills: Azure, Databricks, Pyspark, ADF, Synapse, AI/ML , LLM; Python

We are seeking a highly skilled Senior Azure Data Engineer with expertise in Databricks and PySpark to join our dynamic team. This role involves designing, developing, and maintaining scalable data solutions on Azure, leveraging modern data engineering tools and frameworks. Additionally, 20% of the role will focus on integrating and applying Generative AI (GenAI) technologies to enhance data processing, analytics, and automation capabilities.

Key Responsibilities

• Design, develop, and optimize data pipelines and workflows using Azure Data Factory, Azure Databricks, and PySpark.
• Build and maintain scalable, robust, and secure data architectures on Azure cloud platforms.
• Collaborate with data scientists, analysts, and business stakeholders to integrate data engineering solutions with GenAI models and applications.
• Develop and implement ETL/ELT processes to ingest, transform, and load large volumes of structured and unstructured data.
• Monitor and troubleshoot data pipeline performance, ensuring data quality, integrity, and availability.
• Utilize Azure Synapse Analytics, Azure Data Lake Storage, and related services to support big data and analytics initiatives.
• Lead efforts to incorporate GenAI technologies (e.g., large language models, AI-driven data augmentation, automated data insights) into existing data workflows.
• Stay current with emerging Azure technologies, Databricks features, PySpark enhancements, and GenAI advancements.
• Mentor junior engineers and contribute to best practices, documentation, and knowledge sharing.

Required Skills and Qualifications

• Proven experience with Azure Data Engineering tools: Azure Databricks, Azure Data Factory, Azure Data Lake Storage, and Azure Synapse Analytics.
• Strong proficiency in PySpark and Python for building and optimizing large-scale data pipelines.
• Solid understanding of Generative AI concepts, including experience working with large language models, AI-driven data augmentation, or related GenAI applications.
• Ability to collaborate effectively with AI and data science teams to integrate GenAI models into production data workflows.
• Knowledge of data governance, security best practices, and compliance standards in cloud environments.
• Excellent problem-solving skills and ability to work in an agile, fast-paced environment.
• Strong communication skills for cross-functional collaboration

Key Skills

Education

Any Graduate