Description

You will own the data engineering lifecycle, focusing on Python-based ETL pipelines and migrating infrastructure to Kubernetes.

Responsibilities

• Write and maintain ETL code in Python for data movement and transformations.
• Migrate existing data engineering tasks and infrastructure to Kubernetes.
• Manage and optimize ETL processes on SQL Server.
• Verify table integrity, troubleshoot data issues, and implement corrections.
• Support data science workflows by addressing core data engineering requirements.

Required Skills

• 10+ years of experience in Data Engineering.
• 5+ years of recent or current Python development experience.
• Extensive experience with Apache Spark.
• Proficiency with AWS.
• Strong SQL and MSSQL experience.
• Experience with Apache Airflow.
• Bachelor's degree in Computer Science.
• Ability to pass a live Python coding assessment.

Preferred Skills

• Experience with Azure, Terraform, Scala, Docker, or Kubernetes.
• Background working in tech start-ups.

Key Skills

Education

Bachelor's degree in Computer Science