Description

Build and maintain cloud-native data pipelines and lakehouse foundations using Databricks and AWS services.

This role is on-site.

Responsibilities

• Lead engineering efforts to build pipelines, CDC connections, and lakehouse foundations bridging legacy systems to the cloud.
• Develop comprehensive data pipelines using Databricks Live Tables, PySpark, and AWS Lambda/Kinesis for processing diverse data types.
• Implement Change Data Capture (CDC) using AWS DMS to apply daily delta extracts from Oracle/DB2 to AWS Aurora/PostgreSQL.
• Build transformation frameworks for data validation, cleansing, and business logic within Curated and Foundation layers.
• Expose data products through APIs and event-driven streams for downstream microservices consumption.

Required Skills

• 5+ years of experience with Python, Scala, and PySpark.
• Strong proficiency in SQL and Databricks.
• Experience with Delta Lake architecture.
• Hands-on knowledge of AWS DMS, AWS Lambda, and AWS Kinesis.
• Familiarity with Kafka and event-driven streaming technologies.
• Experience with Oracle or DB2 database systems.

Preferred Skills

• Knowledge of AI Augmentation and Vibe Coding.
• Experience with AWS Aurora or PostgreSQL.
• Background in API development and microservices architecture.

Key Skills

Education

Any Graduate