Optimize AWS EMR clusters and Spark applications to enhance performance, reduce costs, and improve resource utilization.
This role is on-site.
Responsibilities
Analyze EMR cluster metrics, Spark application telemetry, and YARN resource utilization to identify inefficiencies and suboptimal configurations.
Implement cluster-level optimizations including instance right-sizing, node count adjustments, EBS configurations, and spot/on-demand fleet composition.
Tune Spark runtime configurations such as executor memory/core ratios, YARN container sizing, and shuffle service settings to optimize memory utilization.
Validate optimization impact through custom operations, end-to-end deployment, and metric validation using internal tooling.
Develop operational runbooks, SOPs, and programmatic frameworks to scale optimization workflows across hundreds of EMR clusters.
Required Skills
6+ years of experience in cloud computing and data engineering.
Advanced Java concepts and Java Performance Tools (Jprobe, Jensor, OptimizeIT).
Deep expertise in AWS EMR cluster operations and Spark & YARN tuning.
Experience with Memory Optimization and Capacity Analysis.
Proficiency in EC2 Right-Sizing and Cost Optimization strategies.
Strong skills in Monitoring & Troubleshooting using CloudWatch and Logs.