Description

You will build and maintain scalable data ingestion pipelines and scrapers within a cloud-based Lakehouse architecture.

Responsibilities

• Develop scalable Python and PySpark scrapers to ingest structured and unstructured data from HTML, PDF, Excel, APIs, and CSV sources.
• Build and test scrapers in Databricks using established design patterns and shared libraries.
• Orchestrate data workflows using Azure Data Factory (ADF) and integrate with frameworks for logging and error handling.
• Monitor, debug, and maintain data pipelines to ensure reliability and rapid issue resolution.
• Collaborate with Business Analysts and FO Analysts to clarify requirements and validate technical specifications.
• Follow IaC and AZD standards for deployment and infrastructure management.

Required Skills

• 5+ years of experience in data engineering roles.
• Strong expertise in Python programming and SQL.
• Hands-on experience with web scraping and industry best practices.
• Proficiency with PySpark and Big Data frameworks.
• Experience with Databricks and Azure environments.
• Knowledge of Data Lakehouse architectures and Parquet file formats.
• Proficiency with source code management using Git and Azure DevOps.
• Experience working within Agile methodologies like SCRUM or Kanban.
• Competency with JIRA and Confluence for work management.

Preferred Skills

• Familiarity with Python libraries for language translation.
• Certifications in Data Engineering, Azure, or Python.

Key Skills

Education

Any Graduate