Description

You will design and implement high-performance data ingestion pipelines using Apache Spark and Azure Databricks for geospatial datasets.

Responsibilities

• Build scalable, reusable frameworks for ingesting geospatial and imaging data.
• Integrate end-to-end data pipelines from source systems to target repositories while ensuring data quality.
• Develop event-based and streaming technologies for data ingestion and processing.
• Deliver proofs of concept for key technology components to project stakeholders.
• Collaborate with project teams to support API interfaces and search components within Agile and DevOps methodologies.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Direct experience building data pipelines with Azure Data Factory and Apache Spark (preferably Databricks).
• Hands-on experience with Azure Data Analytics components: Azure Storage, Azure SQL Data Warehouse, Azure Data Lake, Azure Cosmos DB, and Azure Stream Analytics.
• Proficiency with Scala/Spark Framework and Azure Databricks.
• Experience with structured and unstructured data, specifically imaging and geospatial data.
• Knowledge of Open Source NoSQL repositories including Snowflake, MongoDB, Cassandra, and Neo4J.
• Experience working in a DevOps environment using tools like Jenkins.
• Strong understanding of Data Management principles and ETL/data warehouse transformation processes.

Preferred Skills

• Microsoft Azure Big Data Architecture certification.
• Experience using geospatial frameworks on Apache Spark.

Key Skills

Education

Bachelor's degree in Computer Science