Description

You design and maintain scalable batch and real-time data ingestion pipelines.

This role is remote.

Responsibilities

• Develop and manage Bronze, Silver, and Gold (Medallion) data layers.
• Implement CDC, watermarking, checkpointing, and batch-to-stream processing.
• Build data quality, validation, reconciliation, and monitoring frameworks.
• Create source-to-canonical mappings and enforce schema versioning and contracts.
• Onboard new data sources and optimize pipeline performance with cross-functional teams.

Required Skills

• 7+ years of experience in Data Engineering or Data Pipeline development.
• Strong expertise with Apache Kafka, including producers, consumers, replay, DLQ, and exactly-once semantics.
• Proficiency in SQL and ETL/ELT fundamentals.
• Hands-on development experience in Java and/or Python.
• Deep understanding of Medallion/Lakehouse architecture.
• Experience implementing data quality and reconciliation frameworks.
• Knowledge of Master Data Management (MDM), identity resolution, and deduplication.
• Proficiency with YAML/JSON configuration and Git.

Preferred Skills

• Experience with probabilistic record matching and Schema Registry (Avro/Protobuf).
• Background in extracting data from legacy or Mainframe systems.

Key Skills

Education

Any Graduate