Description

Key Responsibilities:

Data Pipeline & Orchestration

·      Design, develop, and maintain complex Airflow DAGs for batch and event-driven data pipelines

·      Implement best practices for DAG performance, dependency management, retries, SLA monitoring, and alerting

·      Optimize Airflow scheduler, executor, and worker configurations for high-concurrency workloads

dbt Core & Data Modeling

·      Lead dbt Core implementation, including project structure, environments, and CI/CD integration

·      Design and maintain robust dbt models (staging, intermediate, marts) following analytics engineering best practices

·      Implement dbt tests, documentation, macros, and incremental models to ensure data quality and performance

·      Optimize dbt query performance for large-scale datasets and downstream reporting needs

Cloud, Kubernetes & OpenShift

·      Deploy and manage data workloads on Kubernetes / OpenShift platforms

·      Design strategies for workload distribution, horizontal scaling, and resource optimization

·      Configure CPU/memory requests and limits, autoscaling, and pod scheduling for data workloads

·      Troubleshoot container-level performance issues and resource contention

Performance & Reliability

·      Monitor and tune end-to-end pipeline performance across Airflow, dbt, and data platforms

·      Identify bottlenecks in query execution, orchestration, and infrastructure

·      Implement observability solutions (logs, metrics, alerts) for proactive issue detection

·      Ensure high availability, fault tolerance, and resiliency of data pipelines

Collaboration & Governance

·      Work closely with data architects, platform engineers, and business stakeholders

·      Support financial reporting, accounting, and regulatory data use cases

·      Enforce data engineering standards, security best practices, and governance policies

Required Skills & Qualifications:

Experience

·      10+ years of professional experience in data engineering, analytics engineering, or platform engineering roles

·      Proven experience designing and supporting enterprise-scale data platforms in production environments

Must-Have Technical Skills (Must Reflect in the resume in order to be successful)

·      Expert-level Apache Airflow (DAG design, scheduling, performance tuning)

·      Expert-level DBT Core (data modeling, testing, macros, implementation)

·      Strong proficiency in Python for data engineering and automation

·      Deep understanding of Kubernetes and/or OpenShift in production environments

·      Extensive experience with distributed workload management and performance optimization

·      Strong SQL skills for complex transformations and analytics

Cloud & Platform Experience

·      Experience running data platforms on cloud environments

·      Familiarity with containerized deployments, CI/CD pipelines, and Git-based workflows

Preferred Qualifications

·      Experience supporting financial services or accounting platforms

·      Exposure to enterprise system migrations (e.g., legacy platform to modern data stack)

·      Experience with data warehouses (Oracle)

Key Skills

Education

Bachelor's degree