Description

Design and oversee large-scale data acquisition, modeling, and storage strategies.

Responsibilities

• Define data retention policies and implement ETL/ELT processes with data flow orchestration.
• Scale data pipelines using open-source components and AWS services.
• Monitor system performance and advise on infrastructure changes or capacity planning.
• Mentor junior engineers and collaborate with other architects to deliver integrated solutions.
• Recommend and drive the adoption of new tools and techniques from the big data ecosystem.

Required Skills

• 15+ years of industry experience building and managing big data systems.
• Expertise in building, monitoring, and optimizing cost-efficient pipelines for SaaS.
• Experience with stream-processing systems such as Storm or Spark-Streaming.
• Proficiency in Hadoop, Spark, Kafka, and NoSQL-based data-warehousing solutions.
• Strong knowledge of messaging systems including JMS, Active MQ, Rabbit MQ, and Kafka.
• Hands-on experience with Core Java, Spring/IOC, and design patterns.
• Deep understanding of distributed computing principles and cluster operations.
• Experience integrating SQL and NoSQL databases, file systems, and object storage like S3.
• Competency in big data querying tools such as Pig, Hive, and Impala.

Preferred Skills

• Experience with Web-based SOA architecture implementation.
• Background working with SaaS-based data management products.

Key Skills

Education

Bachelor's Degree