Description

Build and own custom web crawling tools, services, and workflows to improve scrape analysis and reporting.

This role is on-site.

Responsibilities

• Build and maintain custom web crawling tools and services.
• Develop scripts to automate data processing, transforming complex raw data into structured formats.
• Manage end-to-end data ingestion pipelines and ensure data quality through AI-driven processes.
• Design and implement anomaly detection models for geospatial and domain-specific requirements.
• Coordinate specific data requirements with third-party vendors.

Required Skills

• 5+ years of experience with Python, PySpark, SQL, Scala, and Shell scripting.
• Strong proficiency in Python, SQL, C#, and Git.
• Proven experience running large-scale web scrapes and analyzing scraping requirements.
• Experience working as a Data Engineer in a production environment.
• Hands-on experience with SQL and cloud platforms including Azure, AWS, or GCP.
• Deep understanding of data modeling concepts for efficient storage and retrieval systems.
• Background in building anomaly detection models and understanding machine learning algorithms.

Preferred Skills

• Experience with MySQL and distributed notebook environments like Databricks or Azure Synapse.

Key Skills

Education

Any Graduate