Description

You will design and implement scalable data ingestion pipelines and enrichment frameworks using Azure and Apache Spark.

This role is on-site.

Responsibilities

• Design and implement data ingestion pipelines from multiple sources using Apache Spark and Azure Databricks.
• Develop scalable, reusable frameworks for ingesting and enriching datasets.
• Integrate end-to-end pipelines from source systems to target repositories while ensuring data quality.
• Work with event-based and streaming technologies to process data.
• Deliver proofs of concept for key technology components to stakeholders.

Required Skills

• 5 to 10 years of total professional experience.
• Direct experience building data pipelines using Azure Data Factory and Apache Spark.
• Hands-on experience with MS SQL RDBMS and SQL query languages.
• Proficiency in ETL and data warehouse transformation processes.
• Experience designing solutions using Azure Storage, Azure SQL Data Warehouse, and Azure Data Lake.
• Strong knowledge of Data Management principles and Data Modeling.
• Experience working in DevOps environments with tools like Terraform and Microsoft Visual Studio Team Services.

Preferred Skills

• Microsoft Azure Big Data Architecture certification.

Key Skills

Education

Any Graduate