Description

Deliver scalable, high-performance data pipelines using Databricks and Spark that meet rigorous departmental OKRs for performance and cost-efficiency

Build fully automated CI/CD workflows within Gitlab to reduce deployment friction and ensure 100% version-controlled data infrastructure

Apply SOLID engineering principles and modular design to create a reusable testing framework that guarantees data pipeline stability and quality

Foster a culture of excellence by leading technical code reviews and mentoring peers to elevate the team's overall software engineering maturity

Integrate AI-driven automation tools to proactively monitor pipeline health and optimize resource allocation across the AWS ecosystem

Facilitate seamless collaboration between Data Science and Product teams to transform experimental models into fault-tolerant production solutions

Skills, Experience and Requirements

 

Core Skills and Competencies (What you’ll bring)

Advanced proficiency in Python and SQL alongside a deep understanding of distributed systems architecture and modern data patterns

Expertise in Databricks, Spark, and Delta Lake orchestration to manage large-scale, high-velocity data environments

A strong foundation in DevOps methodologies, specifically regarding infrastructure-as-code and containerization using Docker or Kubernetes

AI Application literacy, with the ability to leverage machine learning libraries and NLP frameworks to enhance data processing capabilities

Proven capability in applying design patterns and testing frameworks to ensure the integrity of complex software ecosystems

Critical experience in building and managing highly available, fault-tolerant systems within an enterprise AWS environment

Background in handling sensitive data and maintaining strict security protocols

Minimum Requirements

Minimum Education: Bachelor’s Degree in Computer Science, Electrical Engineering, or a related field

Minimum Experience: 3+ years of experience in data engineering

Required Technical Skills: Must have at least 3+ years of experience with:

Python and SQL

AI Platforms - Databricks and Spark (including Delta Lake)

CI/CD pipelines and Gitlab workflows

Key Skills

Education

Bachelor's degree