Description

Job Description:

We are seeking a Senior Data Engineer with strong expertise in modern data engineering and working knowledge of MLOps practices to support the Knowledge Navigator platform. This individual will play a critical role in building and maintaining scalable data pipelines that feed downstream analytics and AI/ML use cases.

The ideal candidate is a hands-on engineer who thrives in Azure-based environments, is experienced with large-scale data processing using Databricks, and can operate effectively within established enterprise architectures. This role requires a strong focus on data reliability, pipeline performance, and adherence to enterprise data governance and security standards, while collaborating across engineering, analytics, and AI teams.

Responsibilities

• Design, build, and maintain scalable data ingestion and processing pipelines into Azure Data Lake Storage (ADLS) and Databricks
• Implement batch and/or streaming data pipelines to support analytics and AI/ML workloads
• Develop and optimize data transformations using Spark (PySpark) and SQL within Databricks
• Integrate data from enterprise source systems (e.g., APIs, relational databases, SaaS platforms) into centralized data platforms
• Ensure high data quality, integrity, and availability for downstream consumers
• Monitor, troubleshoot, and optimize pipeline performance, reliability, and cost efficiency
• Support pipeline automation, orchestration, and scheduling using tools such as Azure Data Factory or Synapse Pipelines
• Collaborate with data scientists and ML engineers to support data and model pipeline workflows
• Implement logging, monitoring, and alerting to proactively identify issues in production pipelines
• Adhere to enterprise data governance, security, and access control policies
• Work within established architectural frameworks while contributing improvements and best practices
• Partner with cross-functional teams to ensure stable and scalable data operations

Requirements

• 7+ years of professional experience (post-graduate) in data engineering or related field
• Strong hands-on experience with Azure data services, including:
  • Azure Data Lake Storage (ADLS)
  • Azure Data Factory and/or Synapse Pipelines
• Deep experience with Databricks, including Spark / PySpark development and performance optimization
• Proven experience building and maintaining enterprise-scale data pipelines
• Experience integrating data from diverse enterprise systems (databases, APIs, cloud platforms)
• Solid understanding of data architecture principles, data modeling, and ETL/ELT patterns
• Experience with pipeline orchestration, automation, and monitoring
• Working knowledge of MLOps concepts, including supporting data workflows for model training and deployment
• Strong troubleshooting and performance tuning skills in distributed environments
• Familiarity with data governance, security, and access control frameworks
• Proficiency in Python and SQL
• Strong collaboration and communication skills

Preferred (Highly Desired):

• Experience supporting AI/ML or GenAI platforms and workflows
• Familiarity with CI/CD pipelines for data engineering or ML workflows
• Experience with real-time/streaming data technologies (e.g., Kafka, Azure Event Hubs)

Key Skills

Education

Any Graduate