Description

You will design and implement data architectures and transformation pipelines for large-scale datasets.

This role is remote.

Responsibilities

• Develop and modify computer applications, including Python connector creation for Vector pipeline transformations.
• Design end-to-end technical architectures for Data Lakes, reporting marts, and Elastic Search data marts.
• Manage large datasets by implementing partitioning, memory utilization, replication, and fault tolerance during ingestion.
• Develop Elastic Search indices and expose them as RESTful web services for client consumption.
• Create technical specification, system integration, and user acceptance testing documentation.

Required Skills

• 10+ years of professional experience in software engineering or big data environments.
• Proficiency in Python for data transformation and connector development.
• Hands-on experience with Vector Flow and pipeline implementation.
• Experience managing structured and semi-structured data at scale.
• Expertise in Elastic Search index development and RESTful service exposure.
• Experience using Splunk for data set visualization.
• Knowledge of Data Lake architecture and data governance rules.
• Ability to design systems for load balancing and high availability.

Preferred Skills

• Execute proof-of-concepts for new technologies and participate in high-level design and code reviews.

Key Skills

Education

Any Graduate