Description

You will build, maintain, and optimize data pipelines for a large-scale data platform.

Responsibilities

• Build and optimize data pipelines using Spark and Python.
• Collaborate with team members to develop features and consolidate data assets.
• Partner with business analysts and architects to design technical architectures for enterprise projects.
• Develop libraries to standardize data processing workflows.
• Implement automation for applicable data processes.

Required Skills

• 8+ years of professional experience, including 5+ years in a dedicated data engineering role.
• Strong proficiency in Python and SQL.
• Experience building data pipelines with Spark (PySpark).
• Experience building data processing infrastructure in AWS, including Glue or EMR.
• Hands-on experience with relational datastores, NoSQL datastores, and cloud object stores.
• Proficiency with Python libraries such as Pandas and NumPy.
• Strong analytical, problem-solving, and communication skills.

Preferred Skills

• Experience with infrastructure as code, specifically Terraform.
• Production experience with ACID-compliant formats like Hudi, Iceberg, or Delta Lake.

Key Skills

Education

Not specified