Description

Job Description

• End-to-end Data Engineering Design, Build, and Operate scalable Batch & Streaming Pipelines; ingest, transform, and serve across Lake House and Warehouse Environments; automate with Workflow Orchestration Tools.
• Modelling and storage: Implement Conceptual, Logical, and Physical Data Models; optimize Partitioning, Indexing, File Formats (Parquet, Delta, Iceberg), and Table Design for performance and cost.
• Data quality and governance: Enforce standards for Data Freshness, Data Lineage, SLAs/SLOs, Schema Evolution, and PII Handling; integrate with Enterprise Catalogs and Governance Controls.
• Cloud platform engineering: Configure and tune Cloud Data Services (e.g., Azure Databricks, Synapse, Fabric, ADLS, ADF, Snowflake); implement CI/CD, IaC, Observability, and Cost Optimization Patterns.
• Performance and reliability: Profile workloads, remove bottlenecks, manage Shuffle Skew, Joins, and improve reliability via Retry Idempotency, Checkpointing, and Backpressure Strategies.
• Collaboration and leadership: Provide Technical Guidance to Engineers, review Designs and PRs, and contribute to Standards, Blueprints, and Reusable Accelerators. Partner with Product, Platform, and InfoSec teams to land secure solutions.
• Migration and integration: Lead Data Migration/Integration Initiatives, On-Prem to Cloud Migration, and system decommissioning with Zero/Low Downtime Strategies and robust validation.
• Continuous improvement: Evaluate emerging tools (e.g., dbt, Delta Live Tables, Fabric Dataflows), propose upgrades, and institutionalize best practices

Key Skills

Education

Any Graduate