Description

You will design and maintain data platforms using Python and PySpark on AWS. You own the end-to-end development of data pipelines, ensuring data quality and integrity. You implement automation agents for monitoring and logging within the AWS environment. You manage migrations to modern PySpark environments and collaborate with cross-functional teams to align technical solutions with business requirements.

Responsibilities

• Design and implement scalable data pipelines using PySpark and AWS services.
• Develop and maintain robust data platforms using Python and Spark.
• Manage and execute migrations to PySpark-based environments.
• Implement automation agents for system monitoring, logging, and operational efficiency.
• Ensure data quality and integrity across all integrated platforms.

Required Skills

• 3 years of experience in Big Data development.
• Proficiency in Python and PySpark.
• Hands-on experience with AWS cloud services.
• Strong understanding of Big Data technologies and ecosystem.
• Experience with Spark framework for large-scale data processing.
• Ability to collaborate effectively with cross-functional teams.
• Experience designing solutions based on business requirements.

Key Skills

Education

Not specified