Description

You will design and build scalable, metadata-driven ingestion pipelines for batch and streaming datasets using Microsoft Azure and Big Data technologies.

Responsibilities

• Design and build scalable, metadata-driven ingestion pipelines for batch and streaming datasets.
• Execute high-performance data processing and harmonization for structured and unstructured data.
• Schedule, orchestrate, and validate data pipelines using Azure tools.
• Design exception handling and log monitoring systems for debugging.
• Collaborate with Consulting, Data Science, and App Dev teams to deliver analytical solutions.

Required Skills

• 4 to 9 years of total IT experience with at least 2 years in big data engineering.
• Hands-on experience with Databricks, PySpark, ADLS, and SQL Database.
• Proficiency in SQL and Python, Scala, or Java for writing modular, scalable code.
• Experience with Big Data technologies including Hadoop, Spark, Kafka, Hive, or NiFi.
• Knowledge of file formats such as Delta Lake, Avro, Parquet, JSON, and CSV.
• Practical experience with Git, Jenkins, or Azure DevOps within Agile environments.
• Ability to design and build REST APIs.

Preferred Skills

• Experience working specifically on Data Lake or Lakehouse projects.
• Microsoft Azure Data Engineering (DP-203) or Databricks Certified Developer (DE) certifications.
• Experience supporting BI and Data Science teams with secure data consumption.

Key Skills

Education

Any Graduate