Description

You will build and maintain core data infrastructure and processing pipelines.

Responsibilities

• Build and optimize ETL/ELT pipelines using Azure Data Factory and Databricks, ingesting data into ADLS and Snowflake.
• Develop and tune PySpark jobs, implementing Delta Lake, Autoloader, and performance optimizations.
• Design and manage Snowflake schemas, executing complex queries and tuning performance.
• Implement data quality, lineage, and security frameworks across data operations.
• Maintain CI/CD pipelines using Azure DevOps and partner with data architects.

Required Skills

• 5+ years of professional experience in data engineering.
• Expertise with Azure services, including ADF, Databricks, ADLS, and Key Vault.
• Strong experience with Snowflake, including data modeling and query optimization.
• Proficiency in Python, PySpark, Scala, and advanced SQL.
• Knowledge of data formats like JSON, Parquet, and CSV.
• Experience with Delta Lake, Autoloader, Z-ordering, and partitioning.
• Familiarity with CI/CD workflows, Git, and data security best practices.

Preferred Skills

• Azure Data Engineer and Databricks certifications.

Key Skills

Education

Bachelor's or Master's degrees