Description

Design, build, and optimize scalable data pipelines for massive datasets.

This role is on-site.

Responsibilities

• Develop and maintain high-throughput ETL/ELT pipelines using PySpark and Spark SQL.
• Manage and scale data infrastructure on AWS (EMR, Glue) and Azure (Databricks, Synapse).
• Optimize data storage, partitioning, and indexing in Apache Hive and cloud data lakes.
• Resolve performance bottlenecks in Spark jobs, managing data skew and memory utilization.
• Orchestrate automated data workflows using Apache Airflow or native schedulers.

Required Skills

• 10+ years of experience in data engineering.
• Expert-level proficiency in Python and the PySpark API.
• Strong command of HiveQL and ANSI SQL.
• Hands-on experience with Azure Databricks and AWS EMR.
• Deep understanding of Spark architecture (drivers, executors, DAGs).
• Experience with optimized storage formats (Parquet, ORC, Avro).
• Knowledge of dimensional data modeling (Star/Snowflake schemas).

Preferred Skills

• Experience with CI/CD tools: Git, Jenkins, or Ansible.
• Exposure to NoSQL databases such as Cassandra or HBase.

Key Skills

Education

Any Graduate