Description

You will build and maintain scalable data pipelines and analytical tools within the Azure ecosystem.

Responsibilities

• Develop and maintain scalable data pipelines and ETL processes.
• Manage job creation using configurations, dbx, or Asset Bundles.
• Perform data quality checks and debug errors within production environments.
• Manage deployment pipelines across environments using GitHub pull requests.
• Maintain Delta tables and handle version upgrades and environment setups.

Required Skills

• 5+ years of experience in data engineering roles.
• Proficiency in Python and SQL.
• Hands-on experience with Databricks and Delta tables.
• Strong experience with Microsoft Azure and big data technologies, including data lakes and data warehouses.
• Proficiency in GitHub for source control, including workflows, secrets, and environment variables.
• Experience configuring YAML files for Python setups and CI/CD processes.
• Knowledge of PySpark and data integration patterns.
• Ability to develop analytical tools and programs using Python libraries.

Preferred Skills

• Experience with GitHub Actions, events, jobs, and steps for CI/CD.
• Basic knowledge of Snowflake.
• Experience with test frameworks.

Key Skills

Education

Any Graduate