Description

You will build a next-generation data platform to source and store data from diverse technology systems into a centralized environment for reporting and analytics.

Responsibilities

• Develop Python components for a unified data pipeline framework.
• Establish best practices for the efficient use of Airflow, DBT, and Snowflake.
• Test and deploy data pipeline frameworks using standard testing frameworks and CI/CD tooling.
• Monitor and tune the performance of queries, data loads, and Spark jobs.
• Support QA and UAT phases by performing root cause analysis and resolving verified issues.

Required Skills

• 7+ years of experience in data development and solutions within complex, large-volume data environments.
• 7+ years of SQL and PL/SQL experience, including writing complex queries for data analysis.
• 5+ years of experience developing data pipelines and warehousing solutions using Python and libraries such as Pandas, NumPy, and PySpark.
• 3+ years of experience in hybrid data environments (on-prem and cloud).
• 3+ years of experience developing Airflow DAGs with branching, dynamic task generation, and error handling.
• Hands-on experience with Snowflake and Apache Spark.
• Experience developing pipelines for structured, semi-structured, and unstructured data across RDBMS, NoSQL, and Document DBs.
• Proficiency in performance tuning for SQL queries, Spark jobs, and stored procedures.
• Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field.

Preferred Skills

• Hands-on experience with DBT.
• Understanding of advanced data warehouse concepts such as Factless Fact Tables and Temporal/Bi-Temporal models.

Key Skills

Education

Bachelor's degree