Description

Design and implement data pipelines and assets using modern engineering techniques.

Responsibilities

• Build and maintain data pipelines using PySpark and Python.
• Refactor legacy codebases to improve readability, maintainability, and modernization.
• Apply Test-Driven Development (TDD) and write unit tests to ensure code functionality.
• Debug complex code to resolve performance, concurrency, or logic flaws.
• Optimize code performance using Spark SQL and PySpark.

Required Skills

• 4+ years of hands-on programming experience with Python and PySpark.
• Proficiency with Python frameworks, libraries, and best practices.
• Experience using Boto3 for AWS integration.
• Knowledge of AWS services including S3, EC2, Lambda, Redshift, and CloudFormation.
• Strong experience in code optimization using Spark SQL.
• Working knowledge of Git repository management and code versioning.
• Experience with JFrog Artifactory.
• Minimum of 3 years of dedicated data engineering experience.

Key Skills

Education

4+ years' experience in Python, Pyspark