Description

You will lead the architecture and implementation of scalable data flow pipelines across a complex, multi-source telemetry environment.

Responsibilities

• Design and implement modular ingestion frameworks and template-driven architectures for diverse input types and output destinations.
• Develop schema normalization strategies using the Open Cybersecurity Schema Framework (OCSF) to ensure portable field mapping and validation.
• Build custom data transformations and enrichments using scripting languages while enforcing security controls like SSL/TLS and client authentication.
• Ensure end-to-end data lineage and traceability through metadata tagging, correlation IDs, and change tracking.
• Collaborate with observability teams to integrate pipeline health monitoring, failure logging, and anomaly detection.
• Manage data transformation logic including filtering, dynamic routing, and format conversions for over 100 data sources.

Required Skills

• 5+ years of experience in data engineering.
• Hands-on experience with Apache NiFi.
• Proficiency in Kafka.
• Experience with Snowflake.
• Experience with ADX.
• Experience with Splunk.
• Ability to write custom transformations in Groovy, Python, or JavaScript.
• Degree in any field (Any Graduate).

Key Skills

Education

Any Graduate