Description

You will design and maintain large-scale data infrastructure across GCP and AWS, owning the full ETL lifecycle from ingestion to analysis.

This role is on-site.

Responsibilities

• Develop and automate ETL pipelines using Python, Shell, and PySpark.
• Architect and implement Hadoop-based applications using HDFS, YARN, and MapReduce.
• Optimize Spark jobs for heavy data loads and resource allocation.
• Manage and tune GCP services, specifically BigQuery and DataProc.
• Migrate and optimize data solutions on AWS using EMR, Redshift, Glue, Lambda, Athena, and S3.

Required Skills

• 8+ years of experience in Big Data engineering.
• Strong proficiency in Linux and Shell scripting.
• Deep understanding of Hadoop architecture, including High Availability.
• Expertise in Hive, Python, and SQL.
• Hands-on experience with GCP (BigQuery, DataProc) and AWS (EMR, Redshift, Glue, Lambda, Athena, S3).
• Proven track record of tuning Spark jobs and optimizing resource usage.
• Experience designing cost-effective, optimized data applications.

Preferred Skills

• Experience with Hadoop High Availability configurations.
• Background in cloud cost optimization strategies.

Key Skills

Education

Any Graduate