Description

You will build and maintain data pipelines to move information from internal sources into on-premise MongoDB and Azure-based data lakes and warehouses.

Responsibilities

• Design and develop ETL/ELT processes to load data into on-premise ODS and Azure environments.
• Create APIs, data structures, and schemas for efficient data storage and retrieval.
• Develop and maintain data mappings, definitions, architecture, and flow diagrams.
• Implement data quality validation and testing solutions.
• Optimize database performance and tune queries to ensure efficiency.

Required Skills

• 5+ years of experience in Python.
• 3+ years of continuous experience with Airflow.
• 3+ years of experience with Spark.
• 3+ years of experience with relational databases such as Oracle, SQL Server, or PostgreSQL.
• 3+ years of experience writing SQL code.
• 2+ years of experience with columnar databases like Snowflake or Azure Synapse.
• 2+ years of experience with Azure or AWS.

Preferred Skills

• Experience with Apache NiFi, Kafka, Luigi, or Prefect.
• Proficiency in Java, Scala, or R.

Key Skills

Education

Bachelor's degree in Computer Science