Description

Job Description:

We are seeking a highly skilled Data Engineer with 6–10 years of experience in designing, developing, and maintaining scalable data platforms and pipelines. The ideal candidate will have strong expertise in Python, SQL, ETL/ELT processes, PySpark, Databricks, and cloud technologies. The role involves building modern data solutions, optimizing data workflows, enabling analytics, and supporting AI/GenAI initiatives through robust data engineering practices.

Roles & Responsibilities

• Design, develop, and maintain scalable, reliable, and high-performance data pipelines for batch and near real-time processing.
• Build and optimize ETL/ELT workflows using modern data engineering tools and frameworks.
• Develop data transformation and processing solutions using Python, SQL, PySpark, Apache Spark, and Databricks.
• Design and implement data models, data warehouses, and data lakes to support business intelligence and analytics.
• Develop and manage orchestration workflows using Airflow, Matillion, DBT, or similar workflow management tools.
• Work with cloud platforms such as AWS and Azure to build scalable data solutions.
• Integrate data from multiple structured and unstructured sources while ensuring data quality, consistency, and governance.
• Optimize SQL queries, Spark jobs, and overall data processing performance.
• Build and maintain APIs and backend services using Flask, FastAPI, and RESTful services for data integration.
• Collaborate with cross-functional teams including Data Scientists, Business Analysts, Product Owners, and Software Engineers to deliver end-to-end data solutions.
• Implement data quality checks, monitoring, logging, and alerting mechanisms to ensure reliable data delivery.
• Participate in code reviews, testing, CI/CD implementation, deployment, and production support.
• Utilize DevOps best practices including version control, containerization, and automated deployments.
• Contribute to GenAI initiatives by enabling data pipelines for LLMs, Retrieval-Augmented Generation (RAG), vector databases, and AI-powered applications.
• Mentor junior team members and promote engineering best practices across the organization.
• Stay updated with emerging technologies in cloud computing, data engineering, and AI.

Required Technical Skills

Primary Skills

• Python
• SQL
• ETL/ELT Development
• PySpark
• Apache Spark
• Databricks
• Cloud Services (AWS/Azure)
• Data Warehousing Concepts

Data Engineering

• Apache Airflow
• Matillion
• DBT
• Delta Lake
• Data Pipeline Development
• Batch Processing
• Data Orchestration
• Data Quality Frameworks

Cloud Platforms

• AWS
• Microsoft Azure
• Snowflake

Key Skills

Education

Any Graduate