Description

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Azure, Databricks, Kafka, PySpark, SQL, and Delta Lake to design, develop, optimise, and operate enterprise-grade data platforms and production data pipelines.

The ideal candidate should have a strong understanding of cloud data engineering, batch and streaming pipelines, enterprise data architecture, and production operations. Experience implementing knowledge bases and Retrieval-Augmented Generation (RAG) solutions for agentic AI use cases will be an added advantage.

This client-facing role requires excellent communication, stakeholder management, and independent problem-solving skills.

Key Responsibilities

• Design, develop, optimise, and maintain production-grade batch and streaming data pipelines.
• Build scalable ETL/ELT workflows using Azure, Databricks, PySpark, SQL, Kafka, and Delta Lake.
• Implement data orchestration, validation, error handling, monitoring, and observability.
• Optimise pipeline performance, reliability, scalability, and processing costs.
• Implement data quality checks and troubleshoot production issues.
• Apply enterprise data architecture principles, cloud security, access controls, and data governance practices.
• Manage CI/CD pipelines, release processes, deployments, and production support.
• Collaborate with engineers, architects, product owners, source-system teams, and business stakeholders.
• Translate business and technical requirements into secure, scalable, and maintainable data platform solutions.
• Contribute to knowledge base and RAG implementations supporting agentic AI use cases.
• Maintain technical documentation and participate in technical discussions and design reviews.

Mandatory Technical Skills

1. Microsoft Azure and cloud data platforms.
2. Azure Databricks.
3. Apache Kafka and streaming data concepts.
4. PySpark and Apache Spark.
5. SQL and data transformation/validation.
6. Delta Lake.
7. ETL/ELT pipeline development.
8. Production-grade data pipeline design, orchestration, monitoring, and troubleshooting.
9. Python programming.
10. CI/CD, release management, cloud security, and production operations.

Good to Have: Experience implementing knowledge bases, RAG solutions, and data pipelines supporting agentic AI applications.

Experience and Eligibility Criteria

• 5–8 years of relevant experience in data engineering, data platform engineering, or cloud-scale analytics solution delivery.
• Bachelor’s degree in Computer Science, Engineering, or a related discipline.
• Demonstrated ownership of production data pipelines and platform components.
• Hands-on experience with Python and SQL in production environments.
• Strong understanding of enterprise data architecture, cloud security, and access control.
• At least 2 years of continuous employment with a single organisation.
• Excellent verbal and written communication skills.
• Strong stakeholder management and the ability to work independently.
• Immediate joiners or candidates who can join within 45 days.
• Willingness to work from the office in Hinjewadi, Pune.

Job stability is important. Candidates with frequent job changes or without at least 2 years of continuous employment at one organization should be screened carefully.

Key Skills

Education

Not specified