Description

You will build and manage data pipelines and web scraping workflows to support recommendation engines and data-driven predictions.

Responsibilities

• Develop and maintain web scraping scripts to extract data from external websites.
• Capture and process clickstream and temporal data from internal websites.
• Design and implement data pipelines using Azure Data Factory, Azure Databricks, and other Azure services.
• Create ETL scripts using Python or Scala to clean, preprocess, and store data in Azure Data Lake, Azure SQL Database, and Azure Blob Storage.
• Monitor and optimize scraping and processing workflows for performance and cost-efficiency.

Required Skills

• 3+ years of hands-on experience as a Web Scraping Specialist or Web Scraping Engineer.
• Proficiency in Python and web scraping tools such as Scrapy or BeautifulSoup.
• Strong knowledge of Azure Cloud services, including Azure Data Factory, Azure Databricks, Azure Data Lake, Azure SQL Database, and Azure Blob Storage.
• Proficiency in SQL and experience with Apache Spark.
• Familiarity with machine learning concepts and recommendation engines.
• Bachelor's degree in Computer Science, Data Science, or a related field.

Preferred Skills

• Experience with API integration and data extraction from various sources.
• Experience with Microsoft Fabric and Microsoft Data Platform.
• Knowledge of data visualization tools like Power BI or Tableau.

Key Skills

Education

Bachelor's degree