Description

Job Description

Must Have Technical/Functional Skills

• Strong hands-on experience in Python, SQL and data engineering.

• Experience designing and developing ETL/ELT pipelines for large-scale data processing.

• Strong knowledge of PySpark / Spark for data transformation and processing.

• Experience with AWS data services such as Glue, S3, Redshift and related cloud services.

• Hands-on experience with Apache Airflow or equivalent orchestration tools.

• Strong understanding of data transformation, cleansing, validation, reconciliation and data-quality frameworks.

• Experience working with structured, semi-structured and unstructured data.

• Knowledge of batch and streaming data pipelines, preferably Kafka.

• Experience building data pipelines supporting AI/ML, GenAI, LLM and RAG use cases.

• Experience with CI/CD, Git, Docker and Kubernetes/EKS for production deployments.

• Good understanding of data modeling, performance optimization and production support.

Roles & Responsibilities

Design, develop and maintain scalable end-to-end ETL/ELT and data pipelines.

Build data processing and transformation solutions using Python, SQL and PySpark.

Develop batch and streaming pipelines to integrate data from databases, APIs, files and enterprise source systems.

Implement data-quality checks, validation, reconciliation, deduplication and exception handling.

Optimize data pipelines, SQL queries and Spark workloads for high-volume processing.

Prepare and transform datasets required for AI/ML and Generative AI applications.

Develop data ingestion and preparation pipelines supporting LLM/RAG solutions and vector databases.

Collaborate with AI/ML engineers, application teams and business stakeholders on downstream data requirements.

Implement automated deployment and monitoring through CI/CD and DevOps practices.

Support production deployments, monitoring, troubleshooting and root-cause analysis for data-related issues

Key Skills

Education

Any Graduate