Description

Key Responsibilities

Data Engineering & Architecture

- Design, develop, and implement scalable, high-performance cloud data solutions.

- Build and maintain robust ETL/ELT pipelines for data ingestion, transformation, processing, and integration.

- Design scalable data models based on business and technical requirements.

- Integrate structured and unstructured data from:

  - On-premises databases

  - AWS/cloud platforms

  - External data sources

  - APIs and web services

- Optimize data pipelines for performance, reliability, scalability, and cost efficiency.

- Support Data Scientists, Researchers, Analytics, and Reporting teams with reliable data access and processing capabilities.

AWS Data Platform

- Design, develop, and support AWS-based data platforms and data lakes.

- Hands-on experience with:

  - AWS Glue

  - Amazon EMR

  - Amazon Kinesis

  - Amazon ECS

- Configure, monitor, troubleshoot, and maintain AWS data-processing resources and clusters.

- Monitor resource utilization and implement performance and cost optimizations.

- Support provisioning, configuration, upgrades, migrations, and operational maintenance of AWS resources.

SQL & Database Engineering

- Write, review, and optimize complex SQL queries.

- Perform SQL query optimization and database performance tuning.

- Design and manage scalable data models across different database technologies.

- Work with:

  - Relational databases

  - NoSQL databases

  - Document databases

  - Key-value databases

  - Graph databases

  - Time-series databases

- Support database administration activities including backup/recovery, upgrades, migrations, performance troubleshooting, availability, reliability, and cluster management.

Big Data & Streaming

- Develop distributed data-processing solutions using Apache Spark.

- Build and support real-time/streaming data pipelines using Amazon Kinesis and/or Apache Kafka.

- Handle large-scale data processing, transformation, and ingestion workloads.

- Ensure data pipelines are reliable, scalable, and fault tolerant.

Programming & APIs

- Strong programming experience with Python.

- Experience with Java and/or Scala is an advantage.

- Develop and consume REST APIs and web services.

- Integrate data platforms with internal and external applications.

- Support data-harvesting and data-ingestion mechanisms where required.

DevOps, IaC & CI/CD

- Automate infrastructure provisioning and configuration using Terraform / Infrastructure as Code (IaC).

- Build and maintain CI/CD pipelines for data applications and platform deployments.

- Hands-on experience with Jenkins or equivalent CI/CD tools.

- Use Git for source-code and configuration management.

- Automate deployment, configuration, and operational processes.

- Continuously improve development, testing, and deployment workflows.

Machine Learning & Analytics Enablement

- Support the deployment and operationalization of machine-learning models.

- Enable Data Science and Analytics teams with reliable data pipelines and data-access solutions.

- Support use cases involving:

  - Recommendation engines

  - Personalization

  - Search

  - Predictive analytics

- Understand requirements for productionizing ML and data-science workloads.

Security & Data Governance

- Implement data governance and access-control practices across cloud data platforms.

- Follow AWS and cloud-security best practices.

- Support security-risk reduction, data protection, and appropriate access management.

- Ensure data solutions meet organizational security and compliance requirements.

Collaboration & Documentation

- Work closely with Business Teams, Data Scientists, Data Analysts, Architects, and Engineering teams.

- Translate business and technical requirements into practical data solutions.

- Participate in solution design, technical troubleshooting, and implementation activities.

- Prepare and maintain:

  - Technical documentation

  - Architecture/design specifications

  - Data-flow documentation

  - Operational procedures

- Provide technical support and knowledge transfer to platform users and engineering teams.

Mandatory Technical Skills

AWS

- AWS

- AWS Glue

- Amazon EMR

- Amazon Kinesis

- Amazon ECS

Data Engineering

- ETL/ELT

- Data Pipelines

- Data Integration

- Data Transformation

- Data Processing

- Data Lakes / Data Platforms

- Scalable Data Architecture

Big Data & Streaming

- Apache Spark

- Kafka and/or Kinesis

- Distributed Data Processing

- Streaming/Event Processing

Programming

- Python

- Java/Scala – Advantage

Database & SQL

- Advanced SQL

- SQL Query Optimization

- Database Performance Tuning

- Relational Databases

- NoSQL Databases

- Understanding of Document, Key-Value, Graph, and Time-Series Databases

DevOps & Infrastructure

- Terraform

- Infrastructure as Code (IaC)

- Jenkins

- CI/CD

- Git

- Configuration and Deployment Automation

Integration

- REST APIs

- Web Services

- On-premises and External-System Integration

Good to Have

- Experience with Amazon ECS and containerized workloads.

- Experience with both Apache Kafka and Amazon Kinesis.

- Java or Scala development experience.

- Experience with graph, document, key-value, or time-series databases.

- Experience supporting ML model deployment.

- Knowledge of recommendation and personalization systems.

- Experience with TDD/BDD practices.

- Knowledge of configuration-management tools.

- Experience estimating infrastructure and resource requirements.

- Experience with large-scale enterprise data platforms.

Required Experience

- 7–9 years of experience in Data Engineering / Cloud Data Engineering.

- Strong experience designing and developing cloud-based data models and ETL pipelines.

- Strong hands-on AWS experience.

- Proven experience with AWS Glue, EMR, Spark, and Kinesis.

- Strong SQL and database-engineering capabilities.

- Experience with Infrastructure as Code and CI/CD.

- Ability to troubleshoot performance, availability, and operational issues in data platforms.

- Strong analytical, problem-solving, documentation, and stakeholder-management skills

Key Skills

Education

Any Gradute