Description

You will build and maintain scalable data ingestion pipelines and scrapers within a cloud-based Lakehouse architecture.

This role is on-site.

Responsibilities

• Develop scalable Python and PySpark scrapers to ingest structured and unstructured data from HTML, PDF, Excel, APIs, and CSV sources.
• Build, test, and deploy scrapers in Databricks using established design patterns and shared libraries.
• Orchestrate data workflows using Azure Data Factory (ADF) and integrate robust logging and error handling frameworks.
• Monitor, debug, and maintain data pipelines to ensure reliability and enable rapid issue resolution.
• Collaborate with Business and FO Analysts to clarify requirements and validate technical specifications.

Required Skills

• 5+ years of experience in data engineering roles.
• Strong expertise in Python programming and SQL.
• Hands-on experience with web scraping and industry best practices.
• Proficiency with PySpark and Big Data frameworks.
• Experience with Databricks and Azure environments.
• Knowledge of Data Lakehouse architectures and Parquet file formats.
• Proficiency with source code management using Git and Azure DevOps.
• Experience working within Agile methodologies like SCRUM or Kanban.

Preferred Skills

• Familiarity with Python libraries for language translation.
• Certifications in Data Engineering, Azure, or Python.

Key Skills

Education

Any Graduate