Description

Core Responsibilities

• Data Pipeline Engineering: Develop and execute robust API, MCP (Model Context Protocol), and sFTP data ingestions from core IT and business platforms.
• Databricks ODS Architecture: Design, optimize, and maintain data schemas, models, and tables within Databricks using Delta Lake and Medallion architecture (Bronze/Silver/Gold).
• Data Mapping & Lineage: Map source system infrastructure to establish clear technical data lineage from raw application logs to the analytics layer.
• Workflow Automation: Program and manage automated orchestration schedules using Databricks Workflows, Apache Airflow, or cron-based pipelines.
• Performance Reporting Support: Deliver clean, structured, and highly performance data layers optimized for end-user BI dashboard consumption (e.g., Power BI, Tableau).

Technical Qualifications

• Data Platforms: 3+ years of hands-on experience building production data pipelines in Databricks using PySpark, Scala, or Advanced SQL.
• Integration Techniques: Proven expertise developing custom REST API clients, handling sFTP secure transfers via script, and utilizing MCP for contextual data synchronization.
• Data Modeling: Strong foundation in data warehouse modeling, schema evolution, and managing operational data stores (ODS).
• Domain Knowledge: Familiarity with IT Service Management (ITSM) telemetry data, application performance monitoring (APM) tools, or core platform log schemas is a plus

Key Skills

Education

Bachelor's degree