Description

You will build and optimize distributed data processing solutions using Python and PySpark.

Responsibilities

• Develop and optimize ETL pipelines using Python, PySpark, and PySpark Notebooks within AWS EMR.
• Design and implement data ingestion, transformation, and processing workflows using IICS jobs.
• Write efficient, scalable Python code to handle large-scale datasets and distributed computing.
• Monitor job performance, troubleshoot failures, and tune queries for cost and performance efficiency.
• Implement CI/CD, testing, and automation best practices in a cloud environment.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Expertise in Python and PySpark.
• Deep understanding of Spark RDDs, DataFrames, and Datasets.
• Experience working with AWS EMR.
• Practical knowledge of CI/CD and testing frameworks.
• Ability to design and manage IICS jobs for data workflows.
• Experience building distributed computing solutions for large-scale datasets.
• Any Graduate degree.

Key Skills

Education

Any Graduate