Description

Position

We are seeking a skilled Data Engineer with strong expertise in Python, PySpark, AWS Glue, Databricks, and Data Lake technologies. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, data lake architectures, and cloud-based analytics solutions that support enterprise reporting, business intelligence, and advanced analytics initiatives.

Key Responsibilities

Design and develop scalable ETL/ELT pipelines using Python, PySpark, AWS Glue, and Databricks.

Build and maintain enterprise Data Lake solutions for structured, semi-structured, and unstructured data.

Develop data ingestion frameworks for batch and near real-time processing.

Create and optimize data transformation workflows to support analytics and reporting requirements.

Collaborate with business analysts, data architects, and stakeholders to understand data requirements.

Implement data quality validations, reconciliation processes, and monitoring solutions.

Optimize Spark jobs and Databricks workloads for performance and cost efficiency.

Develop reusable frameworks, libraries, and best practices for data engineering teams.

Support cloud migration and modernization initiatives.

Troubleshoot and resolve data pipeline failures and performance bottlenecks.

Ensure data security, governance, and compliance standards are maintained.

Required Technical Skills

Programming & Data Processing, Python, PySpark, SQL, Data Modelling, ETL / ELT Development, Cloud & Big Data, AWS Glue, AWS S3, AWS Lambda, AWS EMR (Preferred)

AWS Redshift (Preferred)

AWS IAM

Databricks

Databricks Workspace

Delta Lake

Spark Structured Streaming

Unity Catalog

Job Orchestration

Performance Optimization

Data Lake Technologies

Data Lake Architecture

Delta Lake

Medallion Architecture (Bronze, Silver, Gold)

Data Governance

Metadata Management

Database Technologies

Snowflake (Preferred)

PostgreSQL

SQL Server

Oracle

DevOps & CI/CD

Git

Azure DevOps / Jenkins

CI/CD Pipelines

Terraform (Preferred)

Preferred Qualifications

Experience with Healthcare, Insurance, Financial Services, or Retail domain.

Hands-on experience in cloud migration projects.

Experience with real-time data processing using Kafka or Kinesis.

Knowledge of Data Governance and Master Data Management.

Understanding of Data Warehousing concepts and dimensional modelling

Key Skills

Education

Any Graduate