Description

You will build and optimize scalable data processing solutions within a data engineering environment.

Responsibilities

• Design, build, and optimize scalable ETL pipelines using PySpark.
• Develop and deploy data processing applications on AWS services including EMR, S3, Lambda, and Glue.
• Perform data transformation, cleansing, and aggregation on large datasets.
• Integrate data processing workflows with other services, using Java for legacy system integration.
• Monitor, troubleshoot, and improve the performance of data jobs in distributed environments.

Required Skills

• 10+ years of professional software development experience.
• 5+ years of hands-on experience with Apache Spark and PySpark.
• Extensive experience with AWS services: EMR, Glue, S3, Lambda, IAM, and CloudWatch.
• Strong understanding of distributed data processing and performance tuning in Spark.
• Proficiency in Python development and writing reusable, efficient code.
• Working knowledge of Java for backend integration and legacy support.
• Experience with SQL for querying large datasets.
• Proficiency with Git version control.
• Familiarity with Agile/Scrum methodologies.

Preferred Skills

• Experience with data structures and algorithms.

Key Skills

Education

Any Gradute