Description

You will design, build, and support big data pipelines and Hadoop ecosystems to process real-time and batch data.

Responsibilities

• Design and implement big data pipelines for real-time data ingestion and processing.
• Develop Hadoop batch jobs to extract data from structured and unstructured sources into Hadoop, Neo4j, MongoDB, and Apache Solr.
• Build near real-time processing solutions using Kafka and Spark Streaming.
• Set up Microsoft Azure with Databricks and Databricks Workspace for business analytics.
• Optimize API performance and implement improvements on slow services to increase responsiveness.
• Automate build processes using Jenkins and Ansible for CI/CD.

Required Skills

• Hadoop
• Kafka
• Spark
• Neo4j
• MongoDB
• Apache Solr
• Spark Streaming and Structured Streaming
• Databricks
• Jenkins and Ansible
• Master’s in Computer Science, Information Technology, or related engineering field
• Minimum 1 year of experience as a Big Data Engineer, Data Engineer, or related role

Key Skills

Education

Bachelor's degree in Computer Science