Description

You will design, build, and maintain scalable data pipelines and frameworks within the Databricks Lakehouse platform.

Responsibilities

• Build and maintain data pipelines to ingest data from source systems into Databricks.
• Design generic frameworks for data ingestion using medallion architecture layers.
• Implement streaming data ingestion using Kafka.
• Develop API-based data integration pipelines and manage CI/CD processes.
• Collaborate with core data platform and data mesh teams to integrate reference architectures.

Required Skills

• 10+ years of experience in data engineering, including OLTP and OLAP database development.
• 5+ years of experience with large-scale data processing using SQL, Python, Pandas, and PySpark.
• 5+ years of experience building data pipelines in Databricks Lakehouse platform.
• Experience implementing medallion architecture for data processing.
• Experience with Kafka for streaming data ingestion.
• Strong skills in data modeling and ER/UML diagramming.
• Experience with orchestration platforms such as Airflow or Tidal.
• Experience with API-based data integration and CI/CD practices.
• Domain knowledge in investment banking and security reference data.
• Proficiency with Jira, Confluence, and ServiceNow.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with Azure Data Factory (ADF) and SQL Server Integration Services (SSIS).

Key Skills

Education

Bachelor's degree