Description

Build and manage large-scale data processing systems and distributed architectures, owning the data pipeline lifecycle from ingestion to retention.

This role is remote.

Responsibilities

• Select and integrate Big Data tools to meet functional requirements.
• Implement ETL processes to ingest data from multiple sources.
• Monitor system performance and advise on necessary infrastructure changes.
• Define and maintain data retention policies.
• Troubleshoot and resolve ongoing issues related to cluster operations.

Required Skills

• 5+ years of experience in big data development.
• Proficiency with Hadoop v2, MapReduce, and HDFS.
• Experience with Spark and building stream-processing systems using Storm or Spark-Streaming.
• Hands-on experience with querying tools including Pig, Hive, and Impala.
• Experience with NoSQL databases such as HBase, Cassandra, or MongoDB.
• Knowledge of ETL techniques and frameworks like Flume.
• Experience with messaging systems such as Kafka or RabbitMQ.
• Familiarity with Big Data ML toolkits like Mahout or SparkML.
• Understanding of Lambda Architecture and its trade-offs.

Preferred Skills

• Experience integrating data from diverse, complex data sources.

Key Skills

Education

Any Graduate