Description

Design, build, and support big data pipelines and Hadoop ecosystems to process real-time and batch data.

Responsibilities

• Design and implement big data pipelines for real-time data ingestion and processing.
• Develop Hadoop batch jobs to extract data from structured and unstructured sources into Hadoop, Neo4j, MongoDB, and Apache Solr.
• Build near real-time processing solutions using Kafka and Spark Streaming.
• Set up Microsoft Azure with Databricks and Databricks Workspace for business analytics.
• Optimize API performance and implement improvements on slow services to increase responsiveness.

Required Skills

• 5+ years of experience as a Big Data Engineer, Data Engineer, or related role.
• Strong proficiency with Hadoop, Kafka, and Spark.
• Experience with Neo4j, MongoDB, and Apache Solr.
• Hands-on experience with Spark Streaming and Structured Streaming.
• Familiarity with Databricks and Azure environments.
• Ability to automate build processes using Jenkins and Ansible for CI/CD.
• Bachelor's degree in Computer Science or related engineering field.

Preferred Skills

• Master’s degree in Computer Science, Information Technology, or related engineering field.

Key Skills

Education

Bachelor's degree in Computer Science