Description

You own the design, construction, and maintenance of scalable data pipelines and ETL/ELT workflows.

Responsibilities

• Develop, construct, test, and maintain data pipelines using Python, PySpark, and SQL.
• Manage and optimize data storage solutions within Snowflake and Data Lakes.
• Integrate data from multiple sources while ensuring data quality, integrity, and security.
• Design and implement automated workflows using Azure Data Factory or similar orchestration tools.
• Monitor and troubleshoot data pipelines, resolving performance bottlenecks and maintaining architecture documentation.

Required Skills

• 10+ years of overall experience in data engineering or a similar role.
• Strong proficiency in Python and PySpark.
• Expertise in SQL and hands-on experience with Snowflake Data Warehouse.
• Experience with data orchestration tools, specifically Azure Data Factory.
• Knowledge of Data Lakes architecture and management principles.
• Experience deploying and managing containerized applications on OpenShift or Kubernetes.
• Familiarity with cloud platforms such as Azure.
• Understanding of data security, governance, and compliance best practices.
• Experience collaborating with Data Scientists and Analysts.

Key Skills

Education

Bachelor's or Master's degrees