Description

You will build and maintain scalable data pipelines to handle large dataset ingestion, transformation, and optimization.

This role is on-site.

Responsibilities

• Design and develop scalable data pipelines for ingesting, transforming, and optimizing large datasets.
• Integrate data from multiple sources into query-optimized formats like Delta Lake, Redshift, and S3.
• Optimize and tune data processing jobs and storage layers for cost efficiency and performance.
• Automate workflows using Airflow, DataBricks APIs, and other orchestration tools.
• Enforce security best practices including IAM, encryption, and compliance within data engineering workflows.

Required Skills

• Strong experience with PySpark and Apache Spark.
• Proficiency in Python and MySQL for data engineering tasks.
• Hands-on experience with AWS services (S3, Redshift, IAM).
• Experience with Azure DataBricks.
• Knowledge of ETL/ELT design, data modeling, and pipeline optimization.
• Familiarity with Airflow or similar orchestration tools.
• Understanding of data security, governance, and compliance standards.

Preferred Skills

• Ability to work effectively in cross-functional teams.

Key Skills

Education

Any Graduate